1Bloodhound [](https://travis-ci.org/bitemyapp/bloodhound) [](https://hackage.haskell.org/package/bloodhound)
4
6Elasticsearch client and query DSL for Haskell
7==============================================
12Search doesn't have to be hard. Let the dog do it.
17"Bloodhound makes Elasticsearch almost tolerable!" - Almost-gruntled user
19"ES is a nightmare but Bloodhound at least makes it tolerable." - Same user, later opinion.
24Elasticsearch \>= 1.0 is recommended. Bloodhound mostly works with 0.9.x, but I don't recommend it if you expect everything to work. As of Bloodhound 0.3 all \>=1.0 versions of Elasticsearch work.
26Current versions we test against are 1.0.3, 1.1.2, 1.2.3, 1.3.2, and 1.4.0. We also check that GHC 7.6 and 7.8 both build and pass tests. See our [TravisCI](https://travis-ci.org/bitemyapp/bloodhound) to learn more.
31Bloodhound is stable for production use. I will strive to avoid breaking API compatibility from here on forward, but dramatic features like a type-safe, fully integrated mapping API may require breaking things in the future.
33Hackage page and Haddock documentation
34======================================
36<http://hackage.haskell.org/package/bloodhound>
41It's not using Bloodhound, but if you need an introduction to or overview of Elasticsearch and how to use it, you can use [this screencast](http://vimeo.com/106463167).
53-- Formatted for use in ghci, so there are "let"s in front of the decls.
55-- if you see :{ and :}, they're so you can copy-paste
56-- the multi-line examples into your ghci REPL.
60import Database.Bloodhound
62import Data.Either (Either(..))
63import Data.Maybe (fromJust)
64import Data.Time.Calendar (Day(..))
65import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
66import Data.Text (Text)
67import GHC.Generics (Generic)
68import Network.HTTP.Client
69import qualified Network.HTTP.Types.Status as NHTS
71-- no trailing slashes in servers, library handles building the path.
72let testServer = (Server "http://localhost:9200")
73let testIndex = IndexName "twitter"
74let testMapping = MappingName "tweet"
76-- defaultIndexSettings is exported by Database.Bloodhound as well
77let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
79-- createIndex returns IO Reply
81-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
82response <- createIndex testServer defaultIndexSettings testIndex
94response <- deleteIndex testServer testIndex
102-- print response if it was a success
103Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
104 , responseVersion = HTTP/1.1
105 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
106 , ("Content-Length", "21")]
107 , responseBody = "{\"acknowledged\":true}"
108 , responseCookieJar = CJ {expose = []}
109 , responseClose' = ResponseClose}
111-- if the index to be deleted didn't exist anyway
112Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
113 , responseVersion = HTTP/1.1
114 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
115 , ("Content-Length","65")]
116 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
117 , responseCookieJar = CJ {expose = []}
118 , responseClose' = ResponseClose}
124#### Note, you **have** to do this if you expect to read what you just wrote
128resp <- refreshIndex testServer testIndex
136-- print resp on success
137Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
138 , responseVersion = HTTP/1.1
139 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
140 , ("Content-Length","50")]
141 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
142 , responseCookieJar = CJ {expose = []}
143 , responseClose' = ResponseClose}
154-- don't forget imports and the like at the top.
156data TweetMapping = TweetMapping deriving (Eq, Show)
158-- I know writing the JSON manually sucks.
159-- I don't have a proper data type for Mappings yet.
160-- Let me know if this is something you need.
163instance ToJSON TweetMapping where
164 toJSON TweetMapping =
166 object ["properties" .=
167 object ["location" .=
168 object ["type" .= ("geo_point" :: Text)]]]]
171resp <- putMapping testServer testIndex testMapping TweetMapping
179resp <- deleteMapping testServer testIndex testMapping
186### Indexing Documents
190-- don't forget the imports and derive generic setting for ghci
191-- at the beginning of the examples.
194data Location = Location { lat :: Double
195 , lon :: Double } deriving (Eq, Generic, Show)
197data Tweet = Tweet { user :: Text
198 , postDate :: UTCTime
201 , location :: Location } deriving (Eq, Generic, Show)
203exampleTweet = Tweet { user = "bitemyapp"
205 (ModifiedJulianDay 55000)
206 (secondsToDiffTime 10)
207 , message = "Use haskell!"
209 , location = Location 40.12 (-71.34) }
211-- automagic (generic) derivation of instances because we're lazy.
213instance FromJSON Tweet
214instance ToJSON Location
215instance FromJSON Location
218-- Should be able to toJSON and encode the data structures like this:
219-- λ> toJSON $ Location 10.0 10.0
220-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
221-- λ> encode $ Location 10.0 10.0
222-- "{\"lat\":10,\"lon\":10}"
224resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
232Response {responseStatus =
233 Status {statusCode = 200, statusMessage = "OK"}
234 , responseVersion = HTTP/1.1, responseHeaders =
235 [("Content-Type","application/json; charset=UTF-8"),
236 ("Content-Length","75")]
237 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
238 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
242### Deleting Documents
246resp <- deleteDocument testServer testIndex testMapping (DocId "1")
254-- n.b., you'll need the earlier imports. responseBody is from http-conduit
256resp <- getDocument testServer testIndex testMapping (DocId "1")
258-- responseBody :: Response body -> body
259let body = responseBody resp
261-- you have two options, you use decode and just get Maybe (EsResult Tweet)
262-- or you can use eitherDecode and get Either String (EsResult Tweet)
264let maybeResult = decode body :: Maybe (EsResult Tweet)
265-- the explicit typing is so Aeson knows how to parse the JSON.
267-- use either if you want to know why something failed to parse.
268-- (string errors, sadly)
269let eitherResult = eitherDecode body :: Either String (EsResult Tweet)
271-- print eitherResult should look like:
272Right (EsResult {_index = "twitter"
277 , _source = Tweet {user = "bitemyapp"
278 , postDate = 2009-06-18 00:00:10 UTC
279 , message = "Use haskell!"
281 , location = Location {lat = 40.12, lon = -71.34}}})
283-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
285-- use the _source record accessor to get at your document
286fmap _source eitherResult
287Right (Tweet {user = "bitemyapp"
288 , postDate = 2009-06-18 00:00:10 UTC
289 , message = "Use haskell!"
291 , location = Location {lat = 40.12, lon = -71.34}})
298### Bulk create, index
302-- don't forget the imports and derive generic setting for ghci
303-- at the beginning of the examples.
306-- Using the earlier Tweet datatype and exampleTweet data
308-- just changing up the data a bit.
309let bulkTest = exampleTweet { user = "blah" }
310let bulkTestTwo = exampleTweet { message = "woohoo!" }
312-- create only bulk operation
313-- BulkCreate :: IndexName -> MappingName -> DocId -> Value -> BulkOperation
314let firstOp = BulkCreate testIndex
315 testMapping (DocId "3") (toJSON bulkTest)
317-- index operation "create or update"
318let sndOp = BulkIndex testIndex
319 testMapping (DocId "4") (toJSON bulkTestTwo)
321-- Some explanation, the final "Value" type that BulkIndex,
322-- BulkCreate, and BulkUpdate accept is the actual document
323-- data that your operation applies to. BulkDelete doesn't
324-- take a value because it's just deleting whatever DocId
327-- list of bulk operations
328let stream = [firstDoc, secondDoc]
330-- Fire off the actual bulk request
331-- bulk :: Server -> [BulkOperation] -> IO Reply
332resp <- bulk testServer stream
337### Encoding individual bulk API operations
340-- the following functions are exported in Bloodhound so
341-- you can build up bulk operations yourself
342encodeBulkOperations :: V.Vector BulkOperation -> L.ByteString
343encodeBulkOperation :: BulkOperation -> L.ByteString
345-- How to use the above:
346data BulkTest = BulkTest { name :: Text } deriving (Eq, Generic, Show)
347instance FromJSON BulkTest
348instance ToJSON BulkTest
351let firstTest = BulkTest "blah"
352let secondTest = BulkTest "bloo"
353let firstDoc = BulkIndex testIndex
354 testMapping (DocId "2") (toJSON firstTest)
355let secondDoc = BulkCreate testIndex
356 testMapping (DocId "3") (toJSON secondTest)
357let stream = V.fromList [firstDoc, secondDoc] :: V.Vector BulkOperation
360let firstDocEncoded = encode firstDoc :: L.ByteString
362-- to encode a vector of bulk operations
363let encodedOperations = encodeBulkOperations stream
365-- to insert into a particular server
366-- bulk :: Server -> V.Vector BulkOperation -> IO Reply
367_ <- bulk testServer stream
380-- exported by the Client module, just defaults some stuff.
381-- mkSearch :: Maybe Query -> Maybe Filter -> Search
382-- mkSearch query filter = Search query filter Nothing False 0 10
384let query = TermQuery (Term "user" "bitemyapp") Nothing
386-- AND'ing identity filter with itself and then tacking it onto a query
387-- search should be a null-operation. I include it for the sake of example.
388-- <||> (or/plus) should make it into a search that returns everything.
390let filter = IdentityFilter <&&> IdentityFilter
392-- constructing the search object the searchByIndex function dispatches on.
393let search = mkSearch (Just query) (Just filter)
395-- you can also searchByType and specify the mapping name.
396reply <- searchByIndex testServer testIndex search
398let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
400λ> fmap (hits . searchHits) result
401Right [Hit {hitIndex = IndexName "twitter"
402 , hitType = MappingName "tweet"
403 , hitDocId = DocId "1"
404 , hitScore = 0.30685282
405 , hitSource = Tweet {user = "bitemyapp"
406 , postDate = 2009-06-18 00:00:10 UTC
407 , message = "Use haskell!"
409 , location = Location {lat = 40.12, lon = -71.34}}}]
417let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
418let search = mkSearch (Just query) Nothing
422#### Multi-Match Query
426let fields = [FieldName "user", FieldName "message"]
427let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
428let search = mkSearch (Just query) Nothing
436let innerQuery = QueryMatchQuery $
437 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
438let query = QueryBoolQuery $
439 mkBoolQuery [innerQuery] [] []
440let search = mkSearch (Just query) Nothing
448let posQuery = QueryMatchQuery $
449 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
450let negQuery = QueryMatchQuery $
451 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
452let query = QueryBoostingQuery $
453 BoostingQuery posQuery negQuery (Boost 0.2)
457#### Rest of the query/filter types
459Just follow the pattern you've seen here and check the Hackage API documentation.
465let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
467-- mkSort is a shortcut function that takes a FieldName and a SortOrder
468-- to generate a vanilla DefaultSort.
469-- checkt the DefaultSort type for the full list of customizable options.
471-- From and size are integers for pagination.
473-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
475-- type Sort = [SortSpec]
476-- type TrackSortScores = Bool
480-- Search takes Maybe Query
486-- just add more sortspecs to the list if you want tie-breakers.
487let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
493#### And, Not, and Or filters
495Filters form a monoid and seminearring.
499instance Monoid Filter where
500 mempty = IdentityFilter
501 mappend a b = AndFilter [a, b] defaultCache
503instance Seminearring Filter where
504 a <||> b = OrFilter [a, b] defaultCache
506-- AndFilter and OrFilter take [Filter] as an argument.
508-- This will return anything, because IdentityFilter returns everything
509OrFilter [IdentityFilter, someOtherFilter] False
511-- This will return exactly what someOtherFilter returns
512AndFilter [IdentityFilter, someOtherFilter] False
514-- Thanks to the seminearring and monoid, the above can be expressed as:
517IdentityFilter <&&> someOtherFilter
520IdentityFilter <||> someOtherFilter
522-- Also there is a NotFilter, it only accepts a single filter, not a list.
524NotFilter someOtherFilter False
532-- And'ing two Identity
533let queryFilter = IdentityFilter <&&> IdentityFilter
535let search = mkSearch Nothing (Just queryFilter)
537reply <- searchByType testServer testIndex testMapping search
543Similar to boolean queries.
547-- Will return only items whose "user" field contains the term "bitemyapp"
548let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
550-- Will return only items whose "user" field does not contain the term "bitemyapp"
551let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
553-- The clause (query) should appear in the matching document.
554-- In a boolean query with no must clauses, one or more should
555-- clauses must match a document. The minimum number of should
556-- clauses to match can be set using the minimum_should_match parameter.
557let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
565-- Will filter for documents that have the field "user"
566let existsFilter = ExistsFilter (FieldName "user")
570#### Geo BoundingBox Filter
574-- topLeft and bottomRight
575let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
577let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False GeoFilterMemory
581#### Geo Distance Filter
585let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
587-- coefficient and units
588let distance = Distance 10.0 Miles
590-- GeoFilterType or NoOptimizeBbox
591let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
593-- SloppyArc is the usual/default optimization in Elasticsearch today
594-- but pre-1.0 versions will need to pick Arc or Plane.
596let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
600#### Geo Distance Range Filter
602Think of a donut and you won't be far off.
606let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
608let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
610let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
614#### Geo Polygon Filter
618-- I think I drew a square here.
619let points = [LatLon 40.0 (-70.00),
620 LatLon 40.0 (-72.00),
621 LatLon 41.0 (-70.00),
622 LatLon 41.0 (-72.00)]
624let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
628#### Document IDs filter
632-- takes a mapping name and a list of DocIds
633IdsFilter (MappingName "tweet") [DocId "1"]
641-- RangeFilter :: FieldName
646let filter = RangeFilter (FieldName "age")
647 (RangeGtLt (GreaterThan 1000.0) (LessThan 100000.0))
648 RangeExecutionIndex False
654let filter = RangeFilter (FieldName "age")
655 (RangeLte (LessThanEq 100000.0))
656 RangeExecutionIndex False
662Date ranges are expressed in UTCTime. Date ranges use the same range bound constructors as numerics, except that they end in "D".
664Note that compatibility with ES is tested only down to seconds.
668let filter = RangeFilter (FieldName "postDate")
670 (GreaterThanD (UTCTime
671 (ModifiedJulianDay 55000)
672 (secondsToDiffTime 9)))
673 (LessThanEqD (UTCTime
674 (ModifiedJulianDay 55000)
675 (secondsToDiffTime 11))))
676 RangeExecutionIndex False
691let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
692 AllRegexpFlags (CacheName "test") False (CacheKey "key")
695-- data RegexpFlags = AllRegexpFlags
697-- | SomeRegexpFlags (NonEmpty RegexpFlag) deriving (Eq, Show)
699-- data RegexpFlag = AnyString
704-- | Interval deriving (Eq, Show)
710#### Adding aggregations to search
712Aggregations can now be added to search queries, or made on their own.
715type Aggregations = M.Map Text Aggregation
717 = TermsAgg TermsAggregation
718 | DateHistogramAgg DateHistogramAggregation
721For convenience, \`\`\`mkAggregations\`\`\` exists, that will create an \`\`\`Aggregations\`\`\` with the aggregation provided.
726 let a = mkAggregations "users" $ TermsAgg $ mkTermsAggregation "user"
727 let search = mkAggregateSearch Nothing a
730Aggregations can be added to an existing search, using the \`\`\`aggBody\`\`\` field
733 let search = mkSearch (Just (MatchAllQuery Nothing)) Nothing
734 let search' = search {aggBody = Just a}
737Since the \`\`\`Aggregations\`\`\` structure is just a Map Text Aggregation, M.insert can be used to add additional aggregations.
740 let a' = M.insert "age" (TermsAgg $ mkTermsAggregation "age") a
743#### Extracting aggregations from results
745Aggregations are part of the reply structure of every search, in the form of
748-- Lift decode and response body to be in the IO monad.
749let decode' = liftM decode
750let responseBody' = liftM responseBody
751let reply = searchByIndex testServer testIndex search
752let response = decode' $ responseBody' reply :: IO (Maybe (SearchResult Tweet))
754-- Now that we have our response, we can extract our terms aggregation result -- which is a list of buckets.
756let terms = do { response' <- response; return $ response' >>= aggregations >>= toTerms "users" }
758Just (Bucket {buckets = [TermsResult {termKey = "bitemyapp", termsDocCount = 1, termsAggs = Nothing}]})
761Note that bucket aggregation results, such as the TermsResult is a member of the type class :
764class BucketAggregation a where
767 aggs :: a -> Maybe AggregationResults
772You can use the function to get any nested results, if there were any. For example, if there were a nested terms aggregation keyed to "age" in a TermsResult named , you would call
774#### Terms Aggregation
778 = TermsAggregation {term :: Either Text Text,
779 termInclude :: Maybe TermInclusion,
780 termExclude :: Maybe TermInclusion,
781 termOrder :: Maybe TermOrder,
782 termMinDocCount :: Maybe Int,
783 termSize :: Maybe Int,
784 termShardSize :: Maybe Int,
785 termCollectMode :: Maybe CollectionMode,
786 termExecutionHint :: Maybe ExecutionHint,
787 termAggs :: Maybe Aggregations}
790Term Aggregations have two factory functions, , and , and can be used as follows:
793let ta = TermsAgg $ mkTermsAggregation "user"
796There are of course other options that can be added to a Terms Aggregation, such as the collection mode:
799let ta = mkTermsAggregation "user"
800let ta' = ta { termCollectMode = Just BreadthFirst }
801let ta'' = TermsAgg ta'
804For more documentation on how the Terms Aggregation works, see <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html>
806#### Date Histogram Aggregation
809data DateHistogramAggregation
810 = DateHistogramAggregation {dateField :: FieldName,
811 dateInterval :: Interval,
812 dateFormat :: Maybe Text,
813 datePreZone :: Maybe Text,
814 datePostZone :: Maybe Text,
815 datePreOffset :: Maybe Text,
816 datePostOffset :: Maybe Text,
817 dateAggs :: Maybe Aggregations}
822The Date Histogram Aggregation works much the same as the Terms Aggregation.
824Relevant functions include , and
827let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") Minute)
830Date histograms also accept a :
833FractionalInterval :: Float -> TimeInterval -> Interval
834-- TimeInterval is the following:
835data TimeInterval = Weeks | Days | Hours | Minutes | Seconds
838It can be used as follows:
841let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") (FractionalInterval 1.5 Minutes))
847data DateHistogramResult
848 = DateHistogramResult {dateKey :: Int,
849 dateKeyStr :: Maybe Text,
851 dateHistogramAggs :: Maybe AggregationResults}
854It is an instance of , and can have nested aggregations in each bucket.
856Buckets can be extracted from a using
858For more information on the Date Histogram Aggregation, see: <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html>
860Possible future functionality
861=============================
866Beginning here: <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html>
871<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html>
873Node discovery and failover
874---------------------------
876Might require TCP support.
878Support for TCP access to Elasticsearch
879---------------------------------------
881Pretend to be a transport client?
883Bulk cluster-join merge
884-----------------------
886Might require making a lucene index on disk with the appropriate format.
891<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html>
896<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html>
901<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html>
906<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html>
911<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html>
916<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html>
921<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html>
926<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting>
928Collapsing redundantly nested and/or structures
929-----------------------------------------------
931The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
933Runtime checking for cycles in data structures
934----------------------------------------------
936check for n \> 1 occurrences in DFS:
938<http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html>
940<http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html>
945Photo from HA! Designs: <https://www.flickr.com/photos/hadesigns/>
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list