3#+CAPTION: Bloodhound (dog)
8[[https://travis-ci.org/bitemyapp/bloodhound][https://travis-ci.org/bitemyapp/bloodhound.svg]]
10#+CAPTION: Hackage version
11[[https://budueba.com/hackage/bloodhound]]
13* Elasticsearch client and query DSL for Haskell
17Search doesn't have to be hard. Let the dog do it.
21"Bloodhound makes Elasticsearch almost tolerable!" - Almost-gruntled user
23"ES is a nightmare but Bloodhound at least makes it tolerable." - Same user, later opinion.
25** Version compatibility
27Elasticsearch >= 1.0 is recommended. Bloodhound mostly works with 0.9.x, but I don't recommend it if you expect everything to work. As of Bloodhound 0.3 all >=1.0 versions of Elasticsearch work.
29Current versions we test against are 1.0.3, 1.1.2, 1.2.3, 1.3.2, and 1.4.0. We also check that GHC 7.6 and 7.8 both build and pass tests. See our [[https://travis-ci.org/bitemyapp/bloodhound][TravisCI]] to learn more.
33Bloodhound is beta at the moment. We've got a solid library tested across multiple ES versions, but coverage of Elasticsearch functionality isn't 100% yet.
35* Hackage page and Haddock documentation
37http://hackage.haskell.org/package/bloodhound
47-- Formatted for use in ghci, so there are "let"s in front of the decls.
49-- if you see :{ and :}, they're so you can copy-paste
50-- the multi-line examples into your ghci REPL.
54import Database.Bloodhound
56import Data.Either (Either(..))
57import Data.Maybe (fromJust)
58import Data.Time.Calendar (Day(..))
59import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
60import Data.Text (Text)
61import GHC.Generics (Generic)
62import Network.HTTP.Conduit
63import qualified Network.HTTP.Types.Status as NHTS
65-- no trailing slashes in servers, library handles building the path.
66let testServer = (Server "http://localhost:9200")
67let testIndex = IndexName "twitter"
68let testMapping = MappingName "tweet"
70-- defaultIndexSettings is exported by Database.Bloodhound as well
71let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
73-- createIndex returns IO Reply
75-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
76response <- createIndex testServer defaultIndexSettings testIndex
88response <- deleteIndex testServer testIndex
96-- print response if it was a success
97Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
98 , responseVersion = HTTP/1.1
99 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
100 , ("Content-Length", "21")]
101 , responseBody = "{\"acknowledged\":true}"
102 , responseCookieJar = CJ {expose = []}
103 , responseClose' = ResponseClose}
105-- if the index to be deleted didn't exist anyway
106Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
107 , responseVersion = HTTP/1.1
108 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
109 , ("Content-Length","65")]
110 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
111 , responseCookieJar = CJ {expose = []}
112 , responseClose' = ResponseClose}
118**** Note, you *have* to do this if you expect to read what you just wrote
122resp <- refreshIndex testServer testIndex
130-- print resp on success
131Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
132 , responseVersion = HTTP/1.1
133 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
134 , ("Content-Length","50")]
135 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
136 , responseCookieJar = CJ {expose = []}
137 , responseClose' = ResponseClose}
147-- don't forget imports and the like at the top.
149data TweetMapping = TweetMapping deriving (Eq, Show)
151-- I know writing the JSON manually sucks.
152-- I don't have a proper data type for Mappings yet.
153-- Let me know if this is something you need.
156instance ToJSON TweetMapping where
157 toJSON TweetMapping =
159 object ["properties" .=
160 object ["location" .=
161 object ["type" .= ("geo_point" :: Text)]]]]
164resp <- putMapping testServer testIndex testMapping TweetMapping
172resp <- deleteMapping testServer testIndex testMapping
176** Document Operations
178*** Indexing Documents
182-- don't forget the imports and derive generic setting for ghci
183-- at the beginning of the examples.
186data Location = Location { lat :: Double
187 , lon :: Double } deriving (Eq, Generic, Show)
189data Tweet = Tweet { user :: Text
190 , postDate :: UTCTime
193 , location :: Location } deriving (Eq, Generic, Show)
195exampleTweet = Tweet { user = "bitemyapp"
197 (ModifiedJulianDay 55000)
198 (secondsToDiffTime 10)
199 , message = "Use haskell!"
201 , location = Location 40.12 (-71.34) }
203-- automagic (generic) derivation of instances because we're lazy.
205instance FromJSON Tweet
206instance ToJSON Location
207instance FromJSON Location
210-- Should be able to toJSON and encode the data structures like this:
211-- λ> toJSON $ Location 10.0 10.0
212-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
213-- λ> encode $ Location 10.0 10.0
214-- "{\"lat\":10,\"lon\":10}"
216resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
224Response {responseStatus =
225 Status {statusCode = 200, statusMessage = "OK"}
226 , responseVersion = HTTP/1.1, responseHeaders =
227 [("Content-Type","application/json; charset=UTF-8"),
228 ("Content-Length","75")]
229 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
230 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
234*** Deleting Documents
238resp <- deleteDocument testServer testIndex testMapping (DocId "1")
246-- n.b., you'll need the earlier imports. responseBody is from http-conduit
248resp <- getDocument testServer testIndex testMapping (DocId "1")
250-- responseBody :: Response body -> body
251let body = responseBody resp
253-- you have two options, you use decode and just get Maybe (EsResult Tweet)
254-- or you can use eitherDecode and get Either String (EsResult Tweet)
256let maybeResult = decode body :: Maybe (EsResult Tweet)
257-- the explicit typing is so Aeson knows how to parse the JSON.
259-- use either if you want to know why something failed to parse.
260-- (string errors, sadly)
261let eitherResult = decode body :: Either String (EsResult Tweet)
263-- print eitherResult should look like:
264Right (EsResult {_index = "twitter"
269 , _source = Tweet {user = "bitemyapp"
270 , postDate = 2009-06-18 00:00:10 UTC
271 , message = "Use haskell!"
273 , location = Location {lat = 40.12, lon = -71.34}}})
275-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
277-- use the _source record accessor to get at your document
278λ> fmap _source result
279Right (Tweet {user = "bitemyapp"
280 , postDate = 2009-06-18 00:00:10 UTC
281 , message = "Use haskell!"
283 , location = Location {lat = 40.12, lon = -71.34}})
289*** Bulk create, index
293-- don't forget the imports and derive generic setting for ghci
294-- at the beginning of the examples.
297-- Using the earlier Tweet datatype and exampleTweet data
299-- just changing up the data a bit.
300let bulkTest = exampleTweet { user = "blah" }
301let bulkTestTwo = exampleTweet { message = "woohoo!" }
303-- create only bulk operation
304-- BulkCreate :: IndexName -> MappingName -> DocId -> Value -> BulkOperation
305let firstOp = BulkCreate testIndex
306 testMapping (DocId "3") (toJSON bulkTest)
308-- index operation "create or update"
309let sndOp = BulkIndex testIndex
310 testMapping (DocId "4") (toJSON bulkTestTwo)
312-- Some explanation, the final "Value" type that BulkIndex,
313-- BulkCreate, and BulkUpdate accept is the actual document
314-- data that your operation applies to. BulkDelete doesn't
315-- take a value because it's just deleting whatever DocId
318-- list of bulk operations
319let stream = [firstDoc, secondDoc]
321-- Fire off the actual bulk request
322-- bulk :: Server -> [BulkOperation] -> IO Reply
323resp <- bulk testServer stream
328*** Encoding individual bulk API operations
331-- the following functions are exported in Bloodhound so
332-- you can build up bulk operations yourself
333encodeBulkOperations :: V.Vector BulkOperation -> L.ByteString
334encodeBulkOperation :: BulkOperation -> L.ByteString
336-- How to use the above:
337data BulkTest = BulkTest { name :: Text } deriving (Eq, Generic, Show)
338instance FromJSON BulkTest
339instance ToJSON BulkTest
342let firstTest = BulkTest "blah"
343let secondTest = BulkTest "bloo"
344let firstDoc = BulkIndex testIndex
345 testMapping (DocId "2") (toJSON firstTest)
346let secondDoc = BulkCreate testIndex
347 testMapping (DocId "3") (toJSON secondTest)
348let stream = V.fromList [firstDoc, secondDoc] :: V.Vector BulkOperation
351let firstDocEncoded = encode firstDoc :: L.ByteString
353-- to encode a vector of bulk operations
354let encodedOperations = encodeBulkOperations stream
356-- to insert into a particular server
357-- bulk :: Server -> V.Vector BulkOperation -> IO Reply
358_ <- bulk testServer stream
370-- exported by the Client module, just defaults some stuff.
371-- mkSearch :: Maybe Query -> Maybe Filter -> Search
372-- mkSearch query filter = Search query filter Nothing False 0 10
374let query = TermQuery (Term "user" "bitemyapp") Nothing
376-- AND'ing identity filter with itself and then tacking it onto a query
377-- search should be a null-operation. I include it for the sake of example.
378-- <||> (or/plus) should make it into a search that returns everything.
380let filter = IdentityFilter <&&> IdentityFilter
382-- constructing the search object the searchByIndex function dispatches on.
383let search = mkSearch (Just query) (Just filter)
385-- you can also searchByType and specify the mapping name.
386reply <- searchByIndex testServer testIndex search
388let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
390λ> fmap (hits . searchHits) result
391Right [Hit {hitIndex = IndexName "twitter"
392 , hitType = MappingName "tweet"
393 , hitDocId = DocId "1"
394 , hitScore = 0.30685282
395 , hitSource = Tweet {user = "bitemyapp"
396 , postDate = 2009-06-18 00:00:10 UTC
397 , message = "Use haskell!"
399 , location = Location {lat = 40.12, lon = -71.34}}}]
407let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
408let search = mkSearch (Just query) Nothing
413**** Multi-Match Query
417let fields = [FieldName "user", FieldName "message"]
418let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
419let search = mkSearch (Just query) Nothing
427let innerQuery = QueryMatchQuery $
428 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
429let query = QueryBoolQuery $
430 mkBoolQuery [innerQuery] [] []
431let search = mkSearch (Just query) Nothing
439let posQuery = QueryMatchQuery $
440 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
441let negQuery = QueryMatchQuery $
442 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
443let query = QueryBoostingQuery $
444 BoostingQuery posQuery negQuery (Boost 0.2)
448**** Rest of the query/filter types
450Just follow the pattern you've seen here and check the Hackage API documentation.
456let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
458-- mkSort is a shortcut function that takes a FieldName and a SortOrder
459-- to generate a vanilla DefaultSort.
460-- checkt the DefaultSort type for the full list of customizable options.
462-- From and size are integers for pagination.
464-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
466-- type Sort = [SortSpec]
467-- type TrackSortScores = Bool
471-- Search takes Maybe Query
477-- just add more sortspecs to the list if you want tie-breakers.
478let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
484**** And, Not, and Or filters
486Filters form a monoid and seminearring.
490instance Monoid Filter where
491 mempty = IdentityFilter
492 mappend a b = AndFilter [a, b] defaultCache
494instance Seminearring Filter where
495 a <||> b = OrFilter [a, b] defaultCache
497-- AndFilter and OrFilter take [Filter] as an argument.
499-- This will return anything, because IdentityFilter returns everything
500OrFilter [IdentityFilter, someOtherFilter] False
502-- This will return exactly what someOtherFilter returns
503AndFilter [IdentityFilter, someOtherFilter] False
505-- Thanks to the seminearring and monoid, the above can be expressed as:
508IdentityFilter <&&> someOtherFilter
511IdentityFilter <||> someOtherFilter
513-- Also there is a NotFilter, it only accepts a single filter, not a list.
515NotFilter someOtherFilter False
523-- And'ing two Identity
524let queryFilter = IdentityFilter <&&> IdentityFilter
526let search = mkSearch Nothing (Just queryFilter)
528reply <- searchByType testServer testIndex testMapping search
534Similar to boolean queries.
538-- Will return only items whose "user" field contains the term "bitemyapp"
539let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
541-- Will return only items whose "user" field does not contain the term "bitemyapp"
542let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
544-- The clause (query) should appear in the matching document.
545-- In a boolean query with no must clauses, one or more should
546-- clauses must match a document. The minimum number of should
547-- clauses to match can be set using the minimum_should_match parameter.
548let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
556-- Will filter for documents that have the field "user"
557let existsFilter = ExistsFilter (FieldName "user")
561**** Geo BoundingBox Filter
565-- topLeft and bottomRight
566let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
568let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False GeoFilterMemory
572**** Geo Distance Filter
576let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
578-- coefficient and units
579let distance = Distance 10.0 Miles
581-- GeoFilterType or NoOptimizeBbox
582let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
584-- SloppyArc is the usual/default optimization in Elasticsearch today
585-- but pre-1.0 versions will need to pick Arc or Plane.
587let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
591**** Geo Distance Range Filter
593Think of a donut and you won't be far off.
597let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
599let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
601let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
605**** Geo Polygon Filter
609-- I think I drew a square here.
610let points = [LatLon 40.0 (-70.00),
611 LatLon 40.0 (-72.00),
612 LatLon 41.0 (-70.00),
613 LatLon 41.0 (-72.00)]
615let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
619**** Document IDs filter
623-- takes a mapping name and a list of DocIds
624IdsFilter (MappingName "tweet") [DocId "1"]
634-- RangeFilter :: FieldName
635-- -> Either HalfRange Range
639let filter = RangeFilter (FieldName "age")
640 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
641 RangeExecutionIndex False
649let filter = RangeFilter (FieldName "age")
650 (Left (HalfRangeLt (LessThan 100000.0)))
651 RangeExecutionIndex False
667let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
668 AllRegexpFlags (CacheName "test") False (CacheKey "key")
671-- data RegexpFlags = AllRegexpFlags
673-- | SomeRegexpFlags (NonEmpty RegexpFlag) deriving (Eq, Show)
675-- data RegexpFlag = AnyString
680-- | Interval deriving (Eq, Show)
685**** Adding aggregations to search
686Aggregations can now be added to search queries, or made on their own.
688type Aggregations = M.Map Text Aggregation
690 = TermsAgg TermsAggregation
691 | DateHistogramAgg DateHistogramAggregation
694For convenience, ```mkAggregations``` exists, that will create an
695```Aggregations``` with the aggregation provided.
699 let a = mkAggregations "users" $ TermsAgg $ mkTermsAggregation "user"
700 let search = mkAggregateSearch Nothing a
703Aggregations can be added to an existing search, using the
707 let search = mkSearch (Just (MatchAllQuery Nothing)) Nothing
708 let search' = search {aggBody = Just a}
711Since the ```Aggregations``` structure is just a Map Text
712Aggregation, M.insert can be used to add additional aggregations.
715 let a' = M.insert "age" (TermsAgg $ mkTermsAggregation "age") a
718**** Extracting aggregations from results
719Aggregations are part of the reply structure of every search, in the
720form of ~Maybe AggregationResults~
723-- Lift decode and response body to be in the IO monad.
724let decode' = liftM decode
725let responseBody' = liftM responseBody
726let reply = searchByIndex testServer testIndex search
727let response = decode' $ responseBody' reply :: IO (Maybe (SearchResult Tweet))
729-- Now that we have our response, we can extract our terms aggregation result -- which is a list of buckets.
731let terms = do { response' <- response; return $ response' >>= aggregations >>= toTerms "users" }
733Just (Bucket {buckets = [TermsResult {termKey = "bitemyapp", termsDocCount = 1, termsAggs = Nothing}]})
736Note that bucket aggregation results, such as the TermsResult is a
737member of the type class ~BucketAggregation~:
740class BucketAggregation a where
743 aggs :: a -> Maybe AggregationResults
746You can use the ~aggs~ function to get any nested results, if
747there were any. For example, if there were a nested terms
748aggregation keyed to "age" in a TermsResult named ~termresult~, you would call ~aggs termresult >>=
751**** Terms Aggregation
754 = TermsAggregation {term :: Either Text Text,
755 termInclude :: Maybe TermInclusion,
756 termExclude :: Maybe TermInclusion,
757 termOrder :: Maybe TermOrder,
758 termMinDocCount :: Maybe Int,
759 termSize :: Maybe Int,
760 termShardSize :: Maybe Int,
761 termCollectMode :: Maybe CollectionMode,
762 termExecutionHint :: Maybe ExecutionHint,
763 termAggs :: Maybe Aggregations}
766Term Aggregations have two factory functions,
767~mkTermsAggregation~, and ~mkTermsScriptAggregation~, and can
771let ta = TermsAgg $ mkTermsAggregation "user"
774There are of course other options that can be added to a Terms
775Aggregation, such as the collection mode:
777let ta = mkTermsAggregation "user"
778let ta' = ta { termCollectMode = Just BreadthFirst }
779let ta'' = TermsAgg ta'
782For more documentation on how the Terms Aggregation works, see
783http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
785**** Date Histogram Aggregation
788data DateHistogramAggregation
789 = DateHistogramAggregation {dateField :: FieldName,
790 dateInterval :: Interval,
791 dateFormat :: Maybe Text,
792 datePreZone :: Maybe Text,
793 datePostZone :: Maybe Text,
794 datePreOffset :: Maybe Text,
795 datePostOffset :: Maybe Text,
796 dateAggs :: Maybe Aggregations}
799The Date Histogram Aggregation works much the same as the Terms
802Relevant functions include ~mkDateHistogram~, and ~toDateHistogram~
805let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") Minute)
808Date histograms also accept a ~FractionalInterval~:
811FractionalInterval :: Float -> TimeInterval -> Interval
812-- TimeInterval is the following:
813data TimeInterval = Weeks | Days | Hours | Minutes | Seconds
816It can be used as follows:
819let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") (FractionalInterval 1.5 Minutes))
822The ~DateHistogramResult~ is defined as:
825data DateHistogramResult
826 = DateHistogramResult {dateKey :: Int,
827 dateKeyStr :: Maybe Text,
829 dateHistogramAggs :: Maybe AggregationResults}
832It is an instance of ~BucketAggregation~, and can have nested
833aggregations in each bucket.
835Buckets can be extracted from a ~AggregationResult~ using
836~toDateHistogram name~
838For more information on the Date Histogram Aggregation, see:
839http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html
841* Possible future functionality
845Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
847** Function Score Query
849http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
851** Node discovery and failover
853Might require TCP support.
855** Support for TCP access to Elasticsearch
857Pretend to be a transport client?
859** Bulk cluster-join merge
861Might require making a lucene index on disk with the appropriate format.
865http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
869http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
871** Geohash cell filter
873http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
877http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
881http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
885http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
889http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
891** Script based sorting
893http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
895** Collapsing redundantly nested and/or structures
897The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
899** Runtime checking for cycles in data structures
901check for n > 1 occurrences in DFS:
903http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
905http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
909Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list