3#+CAPTION: Bloodhound (dog)
8[[https://travis-ci.org/bitemyapp/bloodhound][https://travis-ci.org/bitemyapp/bloodhound.svg]]
10* Elasticsearch client and query DSL for Haskell
14Search doesn't have to be hard. Let the dog do it.
18Bloodhound is alpha at the moment. The library works fine, but I don't want to mislead anyone into thinking the API is final or stable. I wouldn't call the library "complete" or representative of everything you can do in Elasticsearch but compared to clients in other languages, the usability is good.
20* Hackage page and Haddock documentation
22http://hackage.haskell.org/package/bloodhound
32-- Formatted for use in ghci, so there are "let"s in front of the decls.
34-- if you see :{ and :}, they're so you can copy-paste
35-- the multi-line examples into your ghci REPL.
38import Database.Bloodhound
40import Data.Either (Either(..))
41import Data.Maybe (fromJust)
42import Data.Time.Calendar (Day(..))
43import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
44import Data.Text (Text)
45import GHC.Generics (Generic)
46import Network.HTTP.Conduit
47import qualified Network.HTTP.Types.Status as NHTS
49-- no trailing slashes in servers, library handles building the path.
50let testServer = (Server "http://localhost:9200")
51let testIndex = IndexName "twitter"
52let testMapping = MappingName "tweet"
54-- defaultIndexSettings is exported by Database.Bloodhound as well
55let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
57-- createIndex returns IO Reply
59-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
60response <- createIndex testServer defaultIndexSettings testIndex
71response <- deleteIndex testServer testIndex
79-- print response if it was a success
80Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
81 , responseVersion = HTTP/1.1
82 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
83 , ("Content-Length", "21")]
84 , responseBody = "{\"acknowledged\":true}"
85 , responseCookieJar = CJ {expose = []}
86 , responseClose' = ResponseClose}
88-- if the index to be deleted didn't exist anyway
89Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
90 , responseVersion = HTTP/1.1
91 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
92 , ("Content-Length","65")]
93 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
94 , responseCookieJar = CJ {expose = []}
95 , responseClose' = ResponseClose}
101**** Note, you *have* to do this if you expect to read what you just wrote
105resp <- refreshIndex testServer testIndex
113-- print resp on success
114Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
115 , responseVersion = HTTP/1.1
116 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
117 , ("Content-Length","50")]
118 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
119 , responseCookieJar = CJ {expose = []}
120 , responseClose' = ResponseClose}
130-- don't forget imports and the like at the top.
132data TweetMapping = TweetMapping deriving (Eq, Show)
134-- I know writing the JSON manually sucks.
135-- I don't have a proper data type for Mappings yet.
136-- Let me know if this is something you need.
139instance ToJSON TweetMapping where
140 toJSON TweetMapping =
142 object ["properties" .=
143 object ["location" .=
144 object ["type" .= ("geo_point" :: Text)]]]]
147resp <- createMapping testServer testIndex testMapping TweetMapping
155resp <- deleteMapping testServer testIndex testMapping
159** Document Operations
161*** Indexing Documents
165-- don't forget the imports and derive generic setting for ghci
166-- at the beginning of the examples.
169data Location = Location { lat :: Double
170 , lon :: Double } deriving (Eq, Generic, Show)
172data Tweet = Tweet { user :: Text
173 , postDate :: UTCTime
176 , location :: Location } deriving (Eq, Generic, Show)
178exampleTweet = Tweet { user = "bitemyapp"
180 (ModifiedJulianDay 55000)
181 (secondsToDiffTime 10)
182 , message = "Use haskell!"
184 , location = Location 40.12 (-71.34) }
186-- automagic (generic) derivation of instances because we're lazy.
188instance FromJSON Tweet
189instance ToJSON Location
190instance FromJSON Location
193-- Should be able to toJSON and encode the data structures like this:
194-- λ> toJSON $ Location 10.0 10.0
195-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
196-- λ> encode $ Location 10.0 10.0
197-- "{\"lat\":10,\"lon\":10}"
199resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
207Response {responseStatus =
208 Status {statusCode = 200, statusMessage = "OK"}
209 , responseVersion = HTTP/1.1, responseHeaders =
210 [("Content-Type","application/json; charset=UTF-8"),
211 ("Content-Length","75")]
212 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
213 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
217*** Deleting Documents
221resp <- deleteDocument testServer testIndex testMapping (DocId "1")
229-- n.b., you'll need the earlier imports. responseBody is from http-conduit
231resp <- getDocument testServer testIndex testMapping (DocId "1")
233-- responseBody :: Response body -> body
234let body = responseBody resp
236-- you have two options, you use decode and just get Maybe (EsResult Tweet)
237-- or you can use eitherDecode and get Either String (EsResult Tweet)
239let maybeResult = decode body :: Maybe (EsResult Tweet)
240-- the explicit typing is so Aeson knows how to parse the JSON.
242-- use either if you want to know why something failed to parse.
243-- (string errors, sadly)
244let eitherResult = decode body :: Either String (EsResult Tweet)
246-- print eitherResult should look like:
247Right (EsResult {_index = "twitter"
252 , _source = Tweet {user = "bitemyapp"
253 , postDate = 2009-06-18 00:00:10 UTC
254 , message = "Use haskell!"
256 , location = Location {lat = 40.12, lon = -71.34}}})
258-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
260-- use the _source record accessor to get at your document
261λ> fmap _source result
262Right (Tweet {user = "bitemyapp"
263 , postDate = 2009-06-18 00:00:10 UTC
264 , message = "Use haskell!"
266 , location = Location {lat = 40.12, lon = -71.34}})
278-- exported by the Client module, just defaults some stuff.
279-- mkSearch :: Maybe Query -> Maybe Filter -> Search
280-- mkSearch query filter = Search query filter Nothing False 0 10
282let query = TermQuery (Term "user" "bitemyapp") Nothing
284-- AND'ing identity filter with itself and then tacking it onto a query
285-- search should be a null-operation. I include it for the sake of example.
286-- <||> (or/plus) should make it into a search that returns everything.
288let filter = IdentityFilter <&&> IdentityFilter
290-- constructing the search object the searchByIndex function dispatches on.
291let search = mkSearch (Just query) (Just filter)
293-- you can also searchByType and specify the mapping name.
294reply <- searchByIndex testServer testIndex search
296let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
298λ> fmap (hits . searchHits) result
299Right [Hit {hitIndex = IndexName "twitter"
300 , hitType = MappingName "tweet"
301 , hitDocId = DocId "1"
302 , hitScore = 0.30685282
303 , hitSource = Tweet {user = "bitemyapp"
304 , postDate = 2009-06-18 00:00:10 UTC
305 , message = "Use haskell!"
307 , location = Location {lat = 40.12, lon = -71.34}}}]
315let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
316let search = mkSearch (Just query) Nothing
321**** Multi-Match Query
325let fields = [FieldName "user", FieldName "message"]
326let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
327let search = mkSearch (Just query) Nothing
335let innerQuery = QueryMatchQuery $
336 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
337let query = QueryBoolQuery $
338 mkBoolQuery (Just innerQuery) Nothing Nothing
339let search = mkSearch (Just query) Nothing
347let posQuery = QueryMatchQuery $
348 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
349let negQuery = QueryMatchQuery $
350 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
351let query = QueryBoostingQuery $
352 BoostingQuery posQuery negQuery (Boost 0.2)
356**** Rest of the query/filter types
358Just follow the pattern you've seen here and check the Hackage API documentation.
364let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
366-- mkSort is a shortcut function that takes a FieldName and a SortOrder
367-- to generate a vanilla DefaultSort.
368-- checkt the DefaultSort type for the full list of customizable options.
370-- From and size are integers for pagination.
372-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
374-- type Sort = [SortSpec]
375-- type TrackSortScores = Bool
379-- Search takes Maybe Query
385-- just add more sortspecs to the list if you want tie-breakers.
386let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
392**** And, Not, and Or filters
394Filters form a monoid and seminearring.
398instance Monoid Filter where
399 mempty = IdentityFilter
400 mappend a b = AndFilter [a, b] defaultCache
402instance Seminearring Filter where
403 a <||> b = OrFilter [a, b] defaultCache
405-- AndFilter and OrFilter take [Filter] as an argument.
407-- This will return anything, because IdentityFilter returns everything
408OrFilter [IdentityFilter, someOtherFilter] False
410-- This will return exactly what someOtherFilter returns
411AndFilter [IdentityFilter, someOtherFilter] False
413-- Thanks to the seminearring and monoid, the above can be expressed as:
416IdentityFilter <&&> someOtherFilter
419IdentityFilter <||> someOtherFilter
421-- Also there is a NotFilter, it only accepts a single filter, not a list.
423NotFilter someOtherFilter False
431-- And'ing two Identity
432let queryFilter = IdentityFilter <&&> IdentityFilter
434let search = mkSearch Nothing (Just queryFilter)
436reply <- searchByType testServer testIndex testMapping search
442Similar to boolean queries.
446-- Will return only items whose "user" field contains the term "bitemyapp"
447let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
449-- Will return only items whose "user" field does not contain the term "bitemyapp"
450let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
452-- The clause (query) should appear in the matching document.
453-- In a boolean query with no must clauses, one or more should
454-- clauses must match a document. The minimum number of should
455-- clauses to match can be set using the minimum_should_match parameter.
456let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
464-- Will filter for documents that have the field "user"
465let existsFilter = ExistsFilter (FieldName "user")
469**** Geo BoundingBox Filter
473-- topLeft and bottomRight
474let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
476let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False
478-- second argument is GeoFilterType, memory or indexed.
479let geoFilter = GeoBoundingBoxFilter constraint GeoFilterMemory
483**** Geo Distance Filter
487let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
489-- coefficient and units
490let distance = Distance 10.0 Miles
492-- GeoFilterType or NoOptimizeBbox
493let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
495-- SloppyArc is the usual/default optimization in Elasticsearch today
496-- but pre-1.0 versions will need to pick Arc or Plane.
498let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
502**** Geo Distance Range Filter
504Think of a donut and you won't be far off.
508let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
510let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
512let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
516**** Geo Polygon Filter
520-- I think I drew a square here.
521let points = [LatLon 40.0 (-70.00),
522 LatLon 40.0 (-72.00),
523 LatLon 41.0 (-70.00),
524 LatLon 41.0 (-72.00)]
526let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
530**** Document IDs filter
534-- takes a mapping name and a list of DocIds
535IdsFilter (MappingName "tweet") [DocId "1"]
545-- RangeFilter :: FieldName
546-- -> Either HalfRange Range
550let filter = RangeFilter (FieldName "age")
551 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
552 RangeExecutionIndex False
560let filter = RangeFilter (FieldName "age")
561 (Left (HalfRangeLt (LessThan 100000.0)))
562 RangeExecutionIndex False
578let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
579 RegexpAll (CacheName "test") False (CacheKey "key")
581-- RegexpFlags can be a combination of RegexpAll, Complement,
582-- Interval, Intersection, AnyString, and a combination of two options thereof.
586* Possible future functionality
590Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
592** Function Score Query
594http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
596** Node discovery and failover
598Might require TCP support.
600** Support for TCP access to Elasticsearch
602Pretend to be a transport client?
604** Bulk cluster-join merge
606Might require making a lucene index on disk with the appropriate format.
610http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
614http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
616** Geohash cell filter
618http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
622http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
626http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
630http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
634http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
636** Script based sorting
638http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
640** Collapsing redundantly nested and/or structures
642The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
644** Runtime checking for cycles in data structures
646check for n > 1 occurrences in DFS:
648http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
650http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
654Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list