1Bloodhound [](https://travis-ci.org/bitemyapp/bloodhound) [](https://hackage.haskell.org/package/bloodhound)
4
6Elasticsearch client and query DSL for Haskell
7==============================================
12Search doesn't have to be hard. Let the dog do it.
17"Bloodhound makes Elasticsearch almost tolerable!" - Almost-gruntled user
19"ES is a nightmare but Bloodhound at least makes it tolerable." - Same user, later opinion.
24Elasticsearch \>= 1.0 is recommended. Bloodhound mostly works with 0.9.x, but I don't recommend it if you expect everything to work. As of Bloodhound 0.3 all \>=1.0 versions of Elasticsearch work.
26Current versions we test against are 1.0.3, 1.1.2, 1.2.3, 1.3.2, and 1.4.0. We also check that GHC 7.6 and 7.8 both build and pass tests. See our [TravisCI](https://travis-ci.org/bitemyapp/bloodhound) to learn more.
31Bloodhound is stable for production use. I will strive to avoid breaking API compatibility from here on forward, but dramatic features like a type-safe, fully integrated mapping API may require breaking things in the future.
33Hackage page and Haddock documentation
34======================================
36<http://hackage.haskell.org/package/bloodhound>
48-- Formatted for use in ghci, so there are "let"s in front of the decls.
50-- if you see :{ and :}, they're so you can copy-paste
51-- the multi-line examples into your ghci REPL.
55import Database.Bloodhound
57import Data.Either (Either(..))
58import Data.Maybe (fromJust)
59import Data.Time.Calendar (Day(..))
60import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
61import Data.Text (Text)
62import GHC.Generics (Generic)
63import Network.HTTP.Client
64import qualified Network.HTTP.Types.Status as NHTS
66-- no trailing slashes in servers, library handles building the path.
67let testServer = (Server "http://localhost:9200")
68let testIndex = IndexName "twitter"
69let testMapping = MappingName "tweet"
71-- defaultIndexSettings is exported by Database.Bloodhound as well
72let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
74-- createIndex returns IO Reply
76-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
77response <- createIndex testServer defaultIndexSettings testIndex
89response <- deleteIndex testServer testIndex
97-- print response if it was a success
98Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
99 , responseVersion = HTTP/1.1
100 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
101 , ("Content-Length", "21")]
102 , responseBody = "{\"acknowledged\":true}"
103 , responseCookieJar = CJ {expose = []}
104 , responseClose' = ResponseClose}
106-- if the index to be deleted didn't exist anyway
107Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
108 , responseVersion = HTTP/1.1
109 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
110 , ("Content-Length","65")]
111 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
112 , responseCookieJar = CJ {expose = []}
113 , responseClose' = ResponseClose}
119#### Note, you **have** to do this if you expect to read what you just wrote
123resp <- refreshIndex testServer testIndex
131-- print resp on success
132Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
133 , responseVersion = HTTP/1.1
134 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
135 , ("Content-Length","50")]
136 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
137 , responseCookieJar = CJ {expose = []}
138 , responseClose' = ResponseClose}
149-- don't forget imports and the like at the top.
151data TweetMapping = TweetMapping deriving (Eq, Show)
153-- I know writing the JSON manually sucks.
154-- I don't have a proper data type for Mappings yet.
155-- Let me know if this is something you need.
158instance ToJSON TweetMapping where
159 toJSON TweetMapping =
161 object ["properties" .=
162 object ["location" .=
163 object ["type" .= ("geo_point" :: Text)]]]]
166resp <- putMapping testServer testIndex testMapping TweetMapping
174resp <- deleteMapping testServer testIndex testMapping
181### Indexing Documents
185-- don't forget the imports and derive generic setting for ghci
186-- at the beginning of the examples.
189data Location = Location { lat :: Double
190 , lon :: Double } deriving (Eq, Generic, Show)
192data Tweet = Tweet { user :: Text
193 , postDate :: UTCTime
196 , location :: Location } deriving (Eq, Generic, Show)
198exampleTweet = Tweet { user = "bitemyapp"
200 (ModifiedJulianDay 55000)
201 (secondsToDiffTime 10)
202 , message = "Use haskell!"
204 , location = Location 40.12 (-71.34) }
206-- automagic (generic) derivation of instances because we're lazy.
208instance FromJSON Tweet
209instance ToJSON Location
210instance FromJSON Location
213-- Should be able to toJSON and encode the data structures like this:
214-- λ> toJSON $ Location 10.0 10.0
215-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
216-- λ> encode $ Location 10.0 10.0
217-- "{\"lat\":10,\"lon\":10}"
219resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
227Response {responseStatus =
228 Status {statusCode = 200, statusMessage = "OK"}
229 , responseVersion = HTTP/1.1, responseHeaders =
230 [("Content-Type","application/json; charset=UTF-8"),
231 ("Content-Length","75")]
232 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
233 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
237### Deleting Documents
241resp <- deleteDocument testServer testIndex testMapping (DocId "1")
249-- n.b., you'll need the earlier imports. responseBody is from http-conduit
251resp <- getDocument testServer testIndex testMapping (DocId "1")
253-- responseBody :: Response body -> body
254let body = responseBody resp
256-- you have two options, you use decode and just get Maybe (EsResult Tweet)
257-- or you can use eitherDecode and get Either String (EsResult Tweet)
259let maybeResult = decode body :: Maybe (EsResult Tweet)
260-- the explicit typing is so Aeson knows how to parse the JSON.
262-- use either if you want to know why something failed to parse.
263-- (string errors, sadly)
264let eitherResult = eitherDecode body :: Either String (EsResult Tweet)
266-- print eitherResult should look like:
267Right (EsResult {_index = "twitter"
272 , _source = Tweet {user = "bitemyapp"
273 , postDate = 2009-06-18 00:00:10 UTC
274 , message = "Use haskell!"
276 , location = Location {lat = 40.12, lon = -71.34}}})
278-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
280-- use the _source record accessor to get at your document
281fmap _source eitherResult
282Right (Tweet {user = "bitemyapp"
283 , postDate = 2009-06-18 00:00:10 UTC
284 , message = "Use haskell!"
286 , location = Location {lat = 40.12, lon = -71.34}})
293### Bulk create, index
297-- don't forget the imports and derive generic setting for ghci
298-- at the beginning of the examples.
301-- Using the earlier Tweet datatype and exampleTweet data
303-- just changing up the data a bit.
304let bulkTest = exampleTweet { user = "blah" }
305let bulkTestTwo = exampleTweet { message = "woohoo!" }
307-- create only bulk operation
308-- BulkCreate :: IndexName -> MappingName -> DocId -> Value -> BulkOperation
309let firstOp = BulkCreate testIndex
310 testMapping (DocId "3") (toJSON bulkTest)
312-- index operation "create or update"
313let sndOp = BulkIndex testIndex
314 testMapping (DocId "4") (toJSON bulkTestTwo)
316-- Some explanation, the final "Value" type that BulkIndex,
317-- BulkCreate, and BulkUpdate accept is the actual document
318-- data that your operation applies to. BulkDelete doesn't
319-- take a value because it's just deleting whatever DocId
322-- list of bulk operations
323let stream = [firstDoc, secondDoc]
325-- Fire off the actual bulk request
326-- bulk :: Server -> [BulkOperation] -> IO Reply
327resp <- bulk testServer stream
332### Encoding individual bulk API operations
335-- the following functions are exported in Bloodhound so
336-- you can build up bulk operations yourself
337encodeBulkOperations :: V.Vector BulkOperation -> L.ByteString
338encodeBulkOperation :: BulkOperation -> L.ByteString
340-- How to use the above:
341data BulkTest = BulkTest { name :: Text } deriving (Eq, Generic, Show)
342instance FromJSON BulkTest
343instance ToJSON BulkTest
346let firstTest = BulkTest "blah"
347let secondTest = BulkTest "bloo"
348let firstDoc = BulkIndex testIndex
349 testMapping (DocId "2") (toJSON firstTest)
350let secondDoc = BulkCreate testIndex
351 testMapping (DocId "3") (toJSON secondTest)
352let stream = V.fromList [firstDoc, secondDoc] :: V.Vector BulkOperation
355let firstDocEncoded = encode firstDoc :: L.ByteString
357-- to encode a vector of bulk operations
358let encodedOperations = encodeBulkOperations stream
360-- to insert into a particular server
361-- bulk :: Server -> V.Vector BulkOperation -> IO Reply
362_ <- bulk testServer stream
375-- exported by the Client module, just defaults some stuff.
376-- mkSearch :: Maybe Query -> Maybe Filter -> Search
377-- mkSearch query filter = Search query filter Nothing False 0 10
379let query = TermQuery (Term "user" "bitemyapp") Nothing
381-- AND'ing identity filter with itself and then tacking it onto a query
382-- search should be a null-operation. I include it for the sake of example.
383-- <||> (or/plus) should make it into a search that returns everything.
385let filter = IdentityFilter <&&> IdentityFilter
387-- constructing the search object the searchByIndex function dispatches on.
388let search = mkSearch (Just query) (Just filter)
390-- you can also searchByType and specify the mapping name.
391reply <- searchByIndex testServer testIndex search
393let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
395λ> fmap (hits . searchHits) result
396Right [Hit {hitIndex = IndexName "twitter"
397 , hitType = MappingName "tweet"
398 , hitDocId = DocId "1"
399 , hitScore = 0.30685282
400 , hitSource = Tweet {user = "bitemyapp"
401 , postDate = 2009-06-18 00:00:10 UTC
402 , message = "Use haskell!"
404 , location = Location {lat = 40.12, lon = -71.34}}}]
412let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
413let search = mkSearch (Just query) Nothing
417#### Multi-Match Query
421let fields = [FieldName "user", FieldName "message"]
422let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
423let search = mkSearch (Just query) Nothing
431let innerQuery = QueryMatchQuery $
432 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
433let query = QueryBoolQuery $
434 mkBoolQuery [innerQuery] [] []
435let search = mkSearch (Just query) Nothing
443let posQuery = QueryMatchQuery $
444 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
445let negQuery = QueryMatchQuery $
446 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
447let query = QueryBoostingQuery $
448 BoostingQuery posQuery negQuery (Boost 0.2)
452#### Rest of the query/filter types
454Just follow the pattern you've seen here and check the Hackage API documentation.
460let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
462-- mkSort is a shortcut function that takes a FieldName and a SortOrder
463-- to generate a vanilla DefaultSort.
464-- checkt the DefaultSort type for the full list of customizable options.
466-- From and size are integers for pagination.
468-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
470-- type Sort = [SortSpec]
471-- type TrackSortScores = Bool
475-- Search takes Maybe Query
481-- just add more sortspecs to the list if you want tie-breakers.
482let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
488#### And, Not, and Or filters
490Filters form a monoid and seminearring.
494instance Monoid Filter where
495 mempty = IdentityFilter
496 mappend a b = AndFilter [a, b] defaultCache
498instance Seminearring Filter where
499 a <||> b = OrFilter [a, b] defaultCache
501-- AndFilter and OrFilter take [Filter] as an argument.
503-- This will return anything, because IdentityFilter returns everything
504OrFilter [IdentityFilter, someOtherFilter] False
506-- This will return exactly what someOtherFilter returns
507AndFilter [IdentityFilter, someOtherFilter] False
509-- Thanks to the seminearring and monoid, the above can be expressed as:
512IdentityFilter <&&> someOtherFilter
515IdentityFilter <||> someOtherFilter
517-- Also there is a NotFilter, it only accepts a single filter, not a list.
519NotFilter someOtherFilter False
527-- And'ing two Identity
528let queryFilter = IdentityFilter <&&> IdentityFilter
530let search = mkSearch Nothing (Just queryFilter)
532reply <- searchByType testServer testIndex testMapping search
538Similar to boolean queries.
542-- Will return only items whose "user" field contains the term "bitemyapp"
543let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
545-- Will return only items whose "user" field does not contain the term "bitemyapp"
546let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
548-- The clause (query) should appear in the matching document.
549-- In a boolean query with no must clauses, one or more should
550-- clauses must match a document. The minimum number of should
551-- clauses to match can be set using the minimum_should_match parameter.
552let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
560-- Will filter for documents that have the field "user"
561let existsFilter = ExistsFilter (FieldName "user")
565#### Geo BoundingBox Filter
569-- topLeft and bottomRight
570let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
572let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False GeoFilterMemory
576#### Geo Distance Filter
580let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
582-- coefficient and units
583let distance = Distance 10.0 Miles
585-- GeoFilterType or NoOptimizeBbox
586let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
588-- SloppyArc is the usual/default optimization in Elasticsearch today
589-- but pre-1.0 versions will need to pick Arc or Plane.
591let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
595#### Geo Distance Range Filter
597Think of a donut and you won't be far off.
601let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
603let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
605let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
609#### Geo Polygon Filter
613-- I think I drew a square here.
614let points = [LatLon 40.0 (-70.00),
615 LatLon 40.0 (-72.00),
616 LatLon 41.0 (-70.00),
617 LatLon 41.0 (-72.00)]
619let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
623#### Document IDs filter
627-- takes a mapping name and a list of DocIds
628IdsFilter (MappingName "tweet") [DocId "1"]
638-- RangeFilter :: FieldName
639-- -> Either HalfRange Range
643let filter = RangeFilter (FieldName "age")
644 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
645 RangeExecutionIndex False
653let filter = RangeFilter (FieldName "age")
654 (Left (HalfRangeLt (LessThan 100000.0)))
655 RangeExecutionIndex False
671let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
672 AllRegexpFlags (CacheName "test") False (CacheKey "key")
675-- data RegexpFlags = AllRegexpFlags
677-- | SomeRegexpFlags (NonEmpty RegexpFlag) deriving (Eq, Show)
679-- data RegexpFlag = AnyString
684-- | Interval deriving (Eq, Show)
690#### Adding aggregations to search
692Aggregations can now be added to search queries, or made on their own.
695type Aggregations = M.Map Text Aggregation
697 = TermsAgg TermsAggregation
698 | DateHistogramAgg DateHistogramAggregation
701For convenience, \`\`\`mkAggregations\`\`\` exists, that will create an \`\`\`Aggregations\`\`\` with the aggregation provided.
706 let a = mkAggregations "users" $ TermsAgg $ mkTermsAggregation "user"
707 let search = mkAggregateSearch Nothing a
710Aggregations can be added to an existing search, using the \`\`\`aggBody\`\`\` field
713 let search = mkSearch (Just (MatchAllQuery Nothing)) Nothing
714 let search' = search {aggBody = Just a}
717Since the \`\`\`Aggregations\`\`\` structure is just a Map Text Aggregation, M.insert can be used to add additional aggregations.
720 let a' = M.insert "age" (TermsAgg $ mkTermsAggregation "age") a
723#### Extracting aggregations from results
725Aggregations are part of the reply structure of every search, in the form of
728-- Lift decode and response body to be in the IO monad.
729let decode' = liftM decode
730let responseBody' = liftM responseBody
731let reply = searchByIndex testServer testIndex search
732let response = decode' $ responseBody' reply :: IO (Maybe (SearchResult Tweet))
734-- Now that we have our response, we can extract our terms aggregation result -- which is a list of buckets.
736let terms = do { response' <- response; return $ response' >>= aggregations >>= toTerms "users" }
738Just (Bucket {buckets = [TermsResult {termKey = "bitemyapp", termsDocCount = 1, termsAggs = Nothing}]})
741Note that bucket aggregation results, such as the TermsResult is a member of the type class :
744class BucketAggregation a where
747 aggs :: a -> Maybe AggregationResults
752You can use the function to get any nested results, if there were any. For example, if there were a nested terms aggregation keyed to "age" in a TermsResult named , you would call
754#### Terms Aggregation
758 = TermsAggregation {term :: Either Text Text,
759 termInclude :: Maybe TermInclusion,
760 termExclude :: Maybe TermInclusion,
761 termOrder :: Maybe TermOrder,
762 termMinDocCount :: Maybe Int,
763 termSize :: Maybe Int,
764 termShardSize :: Maybe Int,
765 termCollectMode :: Maybe CollectionMode,
766 termExecutionHint :: Maybe ExecutionHint,
767 termAggs :: Maybe Aggregations}
770Term Aggregations have two factory functions, , and , and can be used as follows:
773let ta = TermsAgg $ mkTermsAggregation "user"
776There are of course other options that can be added to a Terms Aggregation, such as the collection mode:
779let ta = mkTermsAggregation "user"
780let ta' = ta { termCollectMode = Just BreadthFirst }
781let ta'' = TermsAgg ta'
784For more documentation on how the Terms Aggregation works, see <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html>
786#### Date Histogram Aggregation
789data DateHistogramAggregation
790 = DateHistogramAggregation {dateField :: FieldName,
791 dateInterval :: Interval,
792 dateFormat :: Maybe Text,
793 datePreZone :: Maybe Text,
794 datePostZone :: Maybe Text,
795 datePreOffset :: Maybe Text,
796 datePostOffset :: Maybe Text,
797 dateAggs :: Maybe Aggregations}
802The Date Histogram Aggregation works much the same as the Terms Aggregation.
804Relevant functions include , and
807let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") Minute)
810Date histograms also accept a :
813FractionalInterval :: Float -> TimeInterval -> Interval
814-- TimeInterval is the following:
815data TimeInterval = Weeks | Days | Hours | Minutes | Seconds
818It can be used as follows:
821let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") (FractionalInterval 1.5 Minutes))
827data DateHistogramResult
828 = DateHistogramResult {dateKey :: Int,
829 dateKeyStr :: Maybe Text,
831 dateHistogramAggs :: Maybe AggregationResults}
834It is an instance of , and can have nested aggregations in each bucket.
836Buckets can be extracted from a using
838For more information on the Date Histogram Aggregation, see: <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html>
840Possible future functionality
841=============================
846Beginning here: <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html>
851<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html>
853Node discovery and failover
854---------------------------
856Might require TCP support.
858Support for TCP access to Elasticsearch
859---------------------------------------
861Pretend to be a transport client?
863Bulk cluster-join merge
864-----------------------
866Might require making a lucene index on disk with the appropriate format.
871<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html>
876<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html>
881<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html>
886<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html>
891<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html>
896<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html>
901<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html>
906<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting>
908Collapsing redundantly nested and/or structures
909-----------------------------------------------
911The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
913Runtime checking for cycles in data structures
914----------------------------------------------
916check for n \> 1 occurrences in DFS:
918<http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html>
920<http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html>
925Photo from HA! Designs: <https://www.flickr.com/photos/hadesigns/>
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list