3#+CAPTION: Bloodhound (dog)
8[[https://travis-ci.org/bitemyapp/bloodhound][https://travis-ci.org/bitemyapp/bloodhound.svg]]
10#+CAPTION: Hackage version
11[[https://hackage.haskell.org/package/bloodhound]]
13* Elasticsearch client and query DSL for Haskell
17Search doesn't have to be hard. Let the dog do it.
21"Bloodhound makes Elasticsearch almost tolerable!" - Almost-gruntled user
23** Version compatibility
25Elasticsearch >= 1.0 is recommended. Bloodhound mostly works with 0.9.x, but I don't recommend it if you expect everything to work. As of Bloodhound 0.3 all >=1.0 versions of Elasticsearch work.
27Current versions we test against are 1.0.3, 1.1.2, 1.2.3, and 1.3.2. We also check that GHC 7.6 and 7.8 both build and pass tests. See our [TravisCI](https://travis-ci.org/bitemyapp/bloodhound) to learn more.
31Bloodhound is beta at the moment. We've got a solid library tested across multiple ES versions, but coverage of Elasticsearch functionality isn't 100% yet.
33* Hackage page and Haddock documentation
35http://hackage.haskell.org/package/bloodhound
45-- Formatted for use in ghci, so there are "let"s in front of the decls.
47-- if you see :{ and :}, they're so you can copy-paste
48-- the multi-line examples into your ghci REPL.
52import Database.Bloodhound
54import Data.Either (Either(..))
55import Data.Maybe (fromJust)
56import Data.Time.Calendar (Day(..))
57import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
58import Data.Text (Text)
59import GHC.Generics (Generic)
60import Network.HTTP.Conduit
61import qualified Network.HTTP.Types.Status as NHTS
63-- no trailing slashes in servers, library handles building the path.
64let testServer = (Server "http://localhost:9200")
65let testIndex = IndexName "twitter"
66let testMapping = MappingName "tweet"
68-- defaultIndexSettings is exported by Database.Bloodhound as well
69let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
71-- createIndex returns IO Reply
73-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
74response <- createIndex testServer defaultIndexSettings testIndex
86response <- deleteIndex testServer testIndex
94-- print response if it was a success
95Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
96 , responseVersion = HTTP/1.1
97 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
98 , ("Content-Length", "21")]
99 , responseBody = "{\"acknowledged\":true}"
100 , responseCookieJar = CJ {expose = []}
101 , responseClose' = ResponseClose}
103-- if the index to be deleted didn't exist anyway
104Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
105 , responseVersion = HTTP/1.1
106 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
107 , ("Content-Length","65")]
108 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
109 , responseCookieJar = CJ {expose = []}
110 , responseClose' = ResponseClose}
116**** Note, you *have* to do this if you expect to read what you just wrote
120resp <- refreshIndex testServer testIndex
128-- print resp on success
129Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
130 , responseVersion = HTTP/1.1
131 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
132 , ("Content-Length","50")]
133 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
134 , responseCookieJar = CJ {expose = []}
135 , responseClose' = ResponseClose}
145-- don't forget imports and the like at the top.
147data TweetMapping = TweetMapping deriving (Eq, Show)
149-- I know writing the JSON manually sucks.
150-- I don't have a proper data type for Mappings yet.
151-- Let me know if this is something you need.
154instance ToJSON TweetMapping where
155 toJSON TweetMapping =
157 object ["properties" .=
158 object ["location" .=
159 object ["type" .= ("geo_point" :: Text)]]]]
162resp <- putMapping testServer testIndex testMapping TweetMapping
170resp <- deleteMapping testServer testIndex testMapping
174** Document Operations
176*** Indexing Documents
180-- don't forget the imports and derive generic setting for ghci
181-- at the beginning of the examples.
184data Location = Location { lat :: Double
185 , lon :: Double } deriving (Eq, Generic, Show)
187data Tweet = Tweet { user :: Text
188 , postDate :: UTCTime
191 , location :: Location } deriving (Eq, Generic, Show)
193exampleTweet = Tweet { user = "bitemyapp"
195 (ModifiedJulianDay 55000)
196 (secondsToDiffTime 10)
197 , message = "Use haskell!"
199 , location = Location 40.12 (-71.34) }
201-- automagic (generic) derivation of instances because we're lazy.
203instance FromJSON Tweet
204instance ToJSON Location
205instance FromJSON Location
208-- Should be able to toJSON and encode the data structures like this:
209-- λ> toJSON $ Location 10.0 10.0
210-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
211-- λ> encode $ Location 10.0 10.0
212-- "{\"lat\":10,\"lon\":10}"
214resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
222Response {responseStatus =
223 Status {statusCode = 200, statusMessage = "OK"}
224 , responseVersion = HTTP/1.1, responseHeaders =
225 [("Content-Type","application/json; charset=UTF-8"),
226 ("Content-Length","75")]
227 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
228 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
232*** Deleting Documents
236resp <- deleteDocument testServer testIndex testMapping (DocId "1")
244-- n.b., you'll need the earlier imports. responseBody is from http-conduit
246resp <- getDocument testServer testIndex testMapping (DocId "1")
248-- responseBody :: Response body -> body
249let body = responseBody resp
251-- you have two options, you use decode and just get Maybe (EsResult Tweet)
252-- or you can use eitherDecode and get Either String (EsResult Tweet)
254let maybeResult = decode body :: Maybe (EsResult Tweet)
255-- the explicit typing is so Aeson knows how to parse the JSON.
257-- use either if you want to know why something failed to parse.
258-- (string errors, sadly)
259let eitherResult = decode body :: Either String (EsResult Tweet)
261-- print eitherResult should look like:
262Right (EsResult {_index = "twitter"
267 , _source = Tweet {user = "bitemyapp"
268 , postDate = 2009-06-18 00:00:10 UTC
269 , message = "Use haskell!"
271 , location = Location {lat = 40.12, lon = -71.34}}})
273-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
275-- use the _source record accessor to get at your document
276λ> fmap _source result
277Right (Tweet {user = "bitemyapp"
278 , postDate = 2009-06-18 00:00:10 UTC
279 , message = "Use haskell!"
281 , location = Location {lat = 40.12, lon = -71.34}})
287*** Bulk create, index
291-- don't forget the imports and derive generic setting for ghci
292-- at the beginning of the examples.
295-- Using the earlier Tweet datatype and exampleTweet data
297-- just changing up the data a bit.
298let bulkTest = exampleTweet { user = "blah" }
299let bulkTestTwo = exampleTweet { message = "woohoo!" }
301-- create only bulk operation
302-- BulkCreate :: IndexName -> MappingName -> DocId -> Value -> BulkOperation
303let firstOp = BulkCreate testIndex
304 testMapping (DocId "3") (toJSON bulkTest)
306-- index operation "create or update"
307let sndOp = BulkIndex testIndex
308 testMapping (DocId "4") (toJSON bulkTestTwo)
310-- Some explanation, the final "Value" type that BulkIndex,
311-- BulkCreate, and BulkUpdate accept is the actual document
312-- data that your operation applies to. BulkDelete doesn't
313-- take a value because it's just deleting whatever DocId
316-- list of bulk operations
317let stream = [firstDoc, secondDoc]
319-- Fire off the actual bulk request
320-- bulk :: Server -> [BulkOperation] -> IO Reply
321resp <- bulk testServer stream
326*** Encoding individual bulk API operations
329-- the following functions are exported in Bloodhound so
330-- you can build up bulk operations yourself
331encodeBulkOperations :: V.Vector BulkOperation -> L.ByteString
332encodeBulkOperation :: BulkOperation -> L.ByteString
334-- How to use the above:
335data BulkTest = BulkTest { name :: Text } deriving (Eq, Generic, Show)
336instance FromJSON BulkTest
337instance ToJSON BulkTest
340let firstTest = BulkTest "blah"
341let secondTest = BulkTest "bloo"
342let firstDoc = BulkIndex testIndex
343 testMapping (DocId "2") (toJSON firstTest)
344let secondDoc = BulkCreate testIndex
345 testMapping (DocId "3") (toJSON secondTest)
346let stream = V.fromList [firstDoc, secondDoc] :: V.Vector BulkOperation
349let firstDocEncoded = encode firstDoc :: L.ByteString
351-- to encode a vector of bulk operations
352let encodedOperations = encodeBulkOperations stream
354-- to insert into a particular server
355-- bulk :: Server -> V.Vector BulkOperation -> IO Reply
356_ <- bulk testServer stream
368-- exported by the Client module, just defaults some stuff.
369-- mkSearch :: Maybe Query -> Maybe Filter -> Search
370-- mkSearch query filter = Search query filter Nothing False 0 10
372let query = TermQuery (Term "user" "bitemyapp") Nothing
374-- AND'ing identity filter with itself and then tacking it onto a query
375-- search should be a null-operation. I include it for the sake of example.
376-- <||> (or/plus) should make it into a search that returns everything.
378let filter = IdentityFilter <&&> IdentityFilter
380-- constructing the search object the searchByIndex function dispatches on.
381let search = mkSearch (Just query) (Just filter)
383-- you can also searchByType and specify the mapping name.
384reply <- searchByIndex testServer testIndex search
386let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
388λ> fmap (hits . searchHits) result
389Right [Hit {hitIndex = IndexName "twitter"
390 , hitType = MappingName "tweet"
391 , hitDocId = DocId "1"
392 , hitScore = 0.30685282
393 , hitSource = Tweet {user = "bitemyapp"
394 , postDate = 2009-06-18 00:00:10 UTC
395 , message = "Use haskell!"
397 , location = Location {lat = 40.12, lon = -71.34}}}]
405let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
406let search = mkSearch (Just query) Nothing
411**** Multi-Match Query
415let fields = [FieldName "user", FieldName "message"]
416let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
417let search = mkSearch (Just query) Nothing
425let innerQuery = QueryMatchQuery $
426 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
427let query = QueryBoolQuery $
428 mkBoolQuery [innerQuery] [] []
429let search = mkSearch (Just query) Nothing
437let posQuery = QueryMatchQuery $
438 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
439let negQuery = QueryMatchQuery $
440 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
441let query = QueryBoostingQuery $
442 BoostingQuery posQuery negQuery (Boost 0.2)
446**** Rest of the query/filter types
448Just follow the pattern you've seen here and check the Hackage API documentation.
454let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
456-- mkSort is a shortcut function that takes a FieldName and a SortOrder
457-- to generate a vanilla DefaultSort.
458-- checkt the DefaultSort type for the full list of customizable options.
460-- From and size are integers for pagination.
462-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
464-- type Sort = [SortSpec]
465-- type TrackSortScores = Bool
469-- Search takes Maybe Query
475-- just add more sortspecs to the list if you want tie-breakers.
476let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
482**** And, Not, and Or filters
484Filters form a monoid and seminearring.
488instance Monoid Filter where
489 mempty = IdentityFilter
490 mappend a b = AndFilter [a, b] defaultCache
492instance Seminearring Filter where
493 a <||> b = OrFilter [a, b] defaultCache
495-- AndFilter and OrFilter take [Filter] as an argument.
497-- This will return anything, because IdentityFilter returns everything
498OrFilter [IdentityFilter, someOtherFilter] False
500-- This will return exactly what someOtherFilter returns
501AndFilter [IdentityFilter, someOtherFilter] False
503-- Thanks to the seminearring and monoid, the above can be expressed as:
506IdentityFilter <&&> someOtherFilter
509IdentityFilter <||> someOtherFilter
511-- Also there is a NotFilter, it only accepts a single filter, not a list.
513NotFilter someOtherFilter False
521-- And'ing two Identity
522let queryFilter = IdentityFilter <&&> IdentityFilter
524let search = mkSearch Nothing (Just queryFilter)
526reply <- searchByType testServer testIndex testMapping search
532Similar to boolean queries.
536-- Will return only items whose "user" field contains the term "bitemyapp"
537let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
539-- Will return only items whose "user" field does not contain the term "bitemyapp"
540let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
542-- The clause (query) should appear in the matching document.
543-- In a boolean query with no must clauses, one or more should
544-- clauses must match a document. The minimum number of should
545-- clauses to match can be set using the minimum_should_match parameter.
546let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
554-- Will filter for documents that have the field "user"
555let existsFilter = ExistsFilter (FieldName "user")
559**** Geo BoundingBox Filter
563-- topLeft and bottomRight
564let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
566let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False GeoFilterMemory
570**** Geo Distance Filter
574let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
576-- coefficient and units
577let distance = Distance 10.0 Miles
579-- GeoFilterType or NoOptimizeBbox
580let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
582-- SloppyArc is the usual/default optimization in Elasticsearch today
583-- but pre-1.0 versions will need to pick Arc or Plane.
585let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
589**** Geo Distance Range Filter
591Think of a donut and you won't be far off.
595let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
597let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
599let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
603**** Geo Polygon Filter
607-- I think I drew a square here.
608let points = [LatLon 40.0 (-70.00),
609 LatLon 40.0 (-72.00),
610 LatLon 41.0 (-70.00),
611 LatLon 41.0 (-72.00)]
613let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
617**** Document IDs filter
621-- takes a mapping name and a list of DocIds
622IdsFilter (MappingName "tweet") [DocId "1"]
632-- RangeFilter :: FieldName
633-- -> Either HalfRange Range
637let filter = RangeFilter (FieldName "age")
638 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
639 RangeExecutionIndex False
647let filter = RangeFilter (FieldName "age")
648 (Left (HalfRangeLt (LessThan 100000.0)))
649 RangeExecutionIndex False
665let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
666 AllRegexpFlags (CacheName "test") False (CacheKey "key")
669-- data RegexpFlags = AllRegexpFlags
671-- | SomeRegexpFlags (NonEmpty RegexpFlag) deriving (Eq, Show)
673-- data RegexpFlag = AnyString
678-- | Interval deriving (Eq, Show)
683**** Adding aggregations to search
684Aggregations can now be added to search queries, or made on their own.
686type Aggregations = M.Map Text Aggregation
688 = TermsAgg TermsAggregation
689 | DateHistogramAgg DateHistogramAggregation
692For convenience, ```mkAggregations``` exists, that will create an
693```Aggregations``` with the aggregation provided.
697 let a = mkAggregations "users" $ TermsAgg $ mkTermsAggregation "user"
698 let search = mkAggregateSearch Nothing a
701Aggregations can be added to an existing search, using the
705 let search = mkSearch (Just (MatchAllQuery Nothing)) Nothing
706 let search' = search {aggBody = Just a}
709Since the ```Aggregations``` structure is just a Map Text
710Aggregation, M.insert can be used to add additional aggregations.
713 let a' = M.insert "age" (TermsAgg $ mkTermsAggregation "age") a
716**** Extracting aggregations from results
717Aggregations are part of the reply structure of every search, in the
718form of ~Maybe AggregationResults~
721-- Lift decode and response body to be in the IO monad.
722let decode' = liftM decode
723let responseBody' = liftM responseBody
724let reply = searchByIndex testServer testIndex search
725let response = decode' $ responseBody' reply :: IO (Maybe (SearchResult Tweet))
727-- Now that we have our response, we can extract our terms aggregation result -- which is a list of buckets.
729let terms = do { response' <- response; return $ response' >>= aggregations >>= toTerms "users" }
731Just (Bucket {buckets = [TermsResult {termKey = "bitemyapp", termsDocCount = 1, termsAggs = Nothing}]})
734Note that bucket aggregation results, such as the TermsResult is a
735member of the type class ~BucketAggregation~:
738class BucketAggregation a where
741 aggs :: a -> Maybe AggregationResults
744You can use the ~aggs~ function to get any nested results, if
745there were any. For example, if there were a nested terms
746aggregation keyed to "age" in a TermsResult named ~termresult~, you would call ~aggs termresult >>=
749**** Terms Aggregation
752 = TermsAggregation {term :: Either Text Text,
753 termInclude :: Maybe TermInclusion,
754 termExclude :: Maybe TermInclusion,
755 termOrder :: Maybe TermOrder,
756 termMinDocCount :: Maybe Int,
757 termSize :: Maybe Int,
758 termShardSize :: Maybe Int,
759 termCollectMode :: Maybe CollectionMode,
760 termExecutionHint :: Maybe ExecutionHint,
761 termAggs :: Maybe Aggregations}
764Term Aggregations have two factory functions,
765~mkTermsAggregation~, and ~mkTermsScriptAggregation~, and can
769let ta = TermsAgg $ mkTermsAggregation "user"
772There are of course other options that can be added to a Terms
773Aggregation, such as the collection mode:
775let ta = mkTermsAggregation "user"
776let ta' = ta { termCollectMode = Just BreadthFirst }
777let ta'' = TermsAgg ta'
780For more documentation on how the Terms Aggregation works, see
781http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
783**** Date Histogram Aggregation
786data DateHistogramAggregation
787 = DateHistogramAggregation {dateField :: FieldName,
788 dateInterval :: Interval,
789 dateFormat :: Maybe Text,
790 datePreZone :: Maybe Text,
791 datePostZone :: Maybe Text,
792 datePreOffset :: Maybe Text,
793 datePostOffset :: Maybe Text,
794 dateAggs :: Maybe Aggregations}
797The Date Histogram Aggregation works much the same as the Terms
800Relevant functions include ~mkDateHistogram~, and ~toDateHistogram~
803let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") Minute)
806Date histograms also accept a ~FractionalInterval~:
809FractionalInterval :: Float -> TimeInterval -> Interval
810-- TimeInterval is the following:
811data TimeInterval = Weeks | Days | Hours | Minutes | Seconds
814It can be used as follows:
817let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") (FractionalInterval 1.5 Minutes))
820The ~DateHistogramResult~ is defined as:
823data DateHistogramResult
824 = DateHistogramResult {dateKey :: Int,
825 dateKeyStr :: Maybe Text,
827 dateHistogramAggs :: Maybe AggregationResults}
830It is an instance of ~BucketAggregation~, and can have nested
831aggregations in each bucket.
833Buckets can be extracted from a ~AggregationResult~ using
834~toDateHistogram name~
836For more information on the Date Histogram Aggregation, see:
837http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html
839* Possible future functionality
843Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
845** Function Score Query
847http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
849** Node discovery and failover
851Might require TCP support.
853** Support for TCP access to Elasticsearch
855Pretend to be a transport client?
857** Bulk cluster-join merge
859Might require making a lucene index on disk with the appropriate format.
863http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
867http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
869** Geohash cell filter
871http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
875http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
879http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
883http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
887http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
889** Script based sorting
891http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
893** Collapsing redundantly nested and/or structures
895The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
897** Runtime checking for cycles in data structures
899check for n > 1 occurrences in DFS:
901http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
903http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
907Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list