3#+CAPTION: Bloodhound (dog)
8[[https://travis-ci.org/bitemyapp/bloodhound][https://travis-ci.org/bitemyapp/bloodhound.svg]]
10* Elasticsearch client and query DSL for Haskell
14Search doesn't have to be hard. Let the dog do it.
16** Version compatibility
18Elasticsearch >= 1.0 is recommended. Bloodhound mostly works with 0.9.x, but I don't recommend it if you expect everything to work. As of Bloodhound 0.3 all >=1.0 versions of Elasticsearch work.
20Current versions we test against are 1.0.3, 1.1.2, 1.2.3, and 1.3.2. We also check that GHC 7.6 and 7.8 both build and pass tests. See our [TravisCI](https://travis-ci.org/bitemyapp/bloodhound) to learn more.
24Bloodhound is beta at the moment. We've got a solid library tested across multiple ES versions, but coverage of Elasticsearch functionality isn't 100% yet.
26* Hackage page and Haddock documentation
28http://hackage.haskell.org/package/bloodhound
38-- Formatted for use in ghci, so there are "let"s in front of the decls.
40-- if you see :{ and :}, they're so you can copy-paste
41-- the multi-line examples into your ghci REPL.
44import Database.Bloodhound
46import Data.Either (Either(..))
47import Data.Maybe (fromJust)
48import Data.Time.Calendar (Day(..))
49import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
50import Data.Text (Text)
51import GHC.Generics (Generic)
52import Network.HTTP.Conduit
53import qualified Network.HTTP.Types.Status as NHTS
55-- no trailing slashes in servers, library handles building the path.
56let testServer = (Server "http://localhost:9200")
57let testIndex = IndexName "twitter"
58let testMapping = MappingName "tweet"
60-- defaultIndexSettings is exported by Database.Bloodhound as well
61let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
63-- createIndex returns IO Reply
65-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
66response <- createIndex testServer defaultIndexSettings testIndex
77response <- deleteIndex testServer testIndex
85-- print response if it was a success
86Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
87 , responseVersion = HTTP/1.1
88 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
89 , ("Content-Length", "21")]
90 , responseBody = "{\"acknowledged\":true}"
91 , responseCookieJar = CJ {expose = []}
92 , responseClose' = ResponseClose}
94-- if the index to be deleted didn't exist anyway
95Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
96 , responseVersion = HTTP/1.1
97 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
98 , ("Content-Length","65")]
99 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
100 , responseCookieJar = CJ {expose = []}
101 , responseClose' = ResponseClose}
107**** Note, you *have* to do this if you expect to read what you just wrote
111resp <- refreshIndex testServer testIndex
119-- print resp on success
120Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
121 , responseVersion = HTTP/1.1
122 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
123 , ("Content-Length","50")]
124 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
125 , responseCookieJar = CJ {expose = []}
126 , responseClose' = ResponseClose}
136-- don't forget imports and the like at the top.
138data TweetMapping = TweetMapping deriving (Eq, Show)
140-- I know writing the JSON manually sucks.
141-- I don't have a proper data type for Mappings yet.
142-- Let me know if this is something you need.
145instance ToJSON TweetMapping where
146 toJSON TweetMapping =
148 object ["properties" .=
149 object ["location" .=
150 object ["type" .= ("geo_point" :: Text)]]]]
153resp <- putMapping testServer testIndex testMapping TweetMapping
161resp <- deleteMapping testServer testIndex testMapping
165** Document Operations
167*** Indexing Documents
171-- don't forget the imports and derive generic setting for ghci
172-- at the beginning of the examples.
175data Location = Location { lat :: Double
176 , lon :: Double } deriving (Eq, Generic, Show)
178data Tweet = Tweet { user :: Text
179 , postDate :: UTCTime
182 , location :: Location } deriving (Eq, Generic, Show)
184exampleTweet = Tweet { user = "bitemyapp"
186 (ModifiedJulianDay 55000)
187 (secondsToDiffTime 10)
188 , message = "Use haskell!"
190 , location = Location 40.12 (-71.34) }
192-- automagic (generic) derivation of instances because we're lazy.
194instance FromJSON Tweet
195instance ToJSON Location
196instance FromJSON Location
199-- Should be able to toJSON and encode the data structures like this:
200-- λ> toJSON $ Location 10.0 10.0
201-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
202-- λ> encode $ Location 10.0 10.0
203-- "{\"lat\":10,\"lon\":10}"
205resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
213Response {responseStatus =
214 Status {statusCode = 200, statusMessage = "OK"}
215 , responseVersion = HTTP/1.1, responseHeaders =
216 [("Content-Type","application/json; charset=UTF-8"),
217 ("Content-Length","75")]
218 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
219 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
223*** Deleting Documents
227resp <- deleteDocument testServer testIndex testMapping (DocId "1")
235-- n.b., you'll need the earlier imports. responseBody is from http-conduit
237resp <- getDocument testServer testIndex testMapping (DocId "1")
239-- responseBody :: Response body -> body
240let body = responseBody resp
242-- you have two options, you use decode and just get Maybe (EsResult Tweet)
243-- or you can use eitherDecode and get Either String (EsResult Tweet)
245let maybeResult = decode body :: Maybe (EsResult Tweet)
246-- the explicit typing is so Aeson knows how to parse the JSON.
248-- use either if you want to know why something failed to parse.
249-- (string errors, sadly)
250let eitherResult = decode body :: Either String (EsResult Tweet)
252-- print eitherResult should look like:
253Right (EsResult {_index = "twitter"
258 , _source = Tweet {user = "bitemyapp"
259 , postDate = 2009-06-18 00:00:10 UTC
260 , message = "Use haskell!"
262 , location = Location {lat = 40.12, lon = -71.34}}})
264-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
266-- use the _source record accessor to get at your document
267λ> fmap _source result
268Right (Tweet {user = "bitemyapp"
269 , postDate = 2009-06-18 00:00:10 UTC
270 , message = "Use haskell!"
272 , location = Location {lat = 40.12, lon = -71.34}})
278*** Bulk create, index
282-- don't forget the imports and derive generic setting for ghci
283-- at the beginning of the examples.
286-- Using the earlier Tweet datatype and exampleTweet data
288-- just changing up the data a bit.
289let bulkTest = exampleTweet { user = "blah" }
290let bulkTestTwo = exampleTweet { message = "woohoo!" }
292-- create only bulk operation
293-- BulkCreate :: IndexName -> MappingName -> DocId -> Value -> BulkOperation
294let firstOp = BulkCreate testIndex
295 testMapping (DocId "3") (toJSON bulkTest)
297-- index operation "create or update"
298let sndOp = BulkIndex testIndex
299 testMapping (DocId "4") (toJSON bulkTestTwo)
301-- Some explanation, the final "Value" type that BulkIndex,
302-- BulkCreate, and BulkUpdate accept is the actual document
303-- data that your operation applies to. BulkDelete doesn't
304-- take a value because it's just deleting whatever DocId
307-- list of bulk operations
308let stream = [firstDoc, secondDoc]
310-- Fire off the actual bulk request
311-- bulk :: Server -> [BulkOperation] -> IO Reply
312resp <- bulk testServer stream
325-- exported by the Client module, just defaults some stuff.
326-- mkSearch :: Maybe Query -> Maybe Filter -> Search
327-- mkSearch query filter = Search query filter Nothing False 0 10
329let query = TermQuery (Term "user" "bitemyapp") Nothing
331-- AND'ing identity filter with itself and then tacking it onto a query
332-- search should be a null-operation. I include it for the sake of example.
333-- <||> (or/plus) should make it into a search that returns everything.
335let filter = IdentityFilter <&&> IdentityFilter
337-- constructing the search object the searchByIndex function dispatches on.
338let search = mkSearch (Just query) (Just filter)
340-- you can also searchByType and specify the mapping name.
341reply <- searchByIndex testServer testIndex search
343let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
345λ> fmap (hits . searchHits) result
346Right [Hit {hitIndex = IndexName "twitter"
347 , hitType = MappingName "tweet"
348 , hitDocId = DocId "1"
349 , hitScore = 0.30685282
350 , hitSource = Tweet {user = "bitemyapp"
351 , postDate = 2009-06-18 00:00:10 UTC
352 , message = "Use haskell!"
354 , location = Location {lat = 40.12, lon = -71.34}}}]
362let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
363let search = mkSearch (Just query) Nothing
368**** Multi-Match Query
372let fields = [FieldName "user", FieldName "message"]
373let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
374let search = mkSearch (Just query) Nothing
382let innerQuery = QueryMatchQuery $
383 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
384let query = QueryBoolQuery $
385 mkBoolQuery (Just innerQuery) Nothing Nothing
386let search = mkSearch (Just query) Nothing
394let posQuery = QueryMatchQuery $
395 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
396let negQuery = QueryMatchQuery $
397 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
398let query = QueryBoostingQuery $
399 BoostingQuery posQuery negQuery (Boost 0.2)
403**** Rest of the query/filter types
405Just follow the pattern you've seen here and check the Hackage API documentation.
411let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
413-- mkSort is a shortcut function that takes a FieldName and a SortOrder
414-- to generate a vanilla DefaultSort.
415-- checkt the DefaultSort type for the full list of customizable options.
417-- From and size are integers for pagination.
419-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
421-- type Sort = [SortSpec]
422-- type TrackSortScores = Bool
426-- Search takes Maybe Query
432-- just add more sortspecs to the list if you want tie-breakers.
433let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
439**** And, Not, and Or filters
441Filters form a monoid and seminearring.
445instance Monoid Filter where
446 mempty = IdentityFilter
447 mappend a b = AndFilter [a, b] defaultCache
449instance Seminearring Filter where
450 a <||> b = OrFilter [a, b] defaultCache
452-- AndFilter and OrFilter take [Filter] as an argument.
454-- This will return anything, because IdentityFilter returns everything
455OrFilter [IdentityFilter, someOtherFilter] False
457-- This will return exactly what someOtherFilter returns
458AndFilter [IdentityFilter, someOtherFilter] False
460-- Thanks to the seminearring and monoid, the above can be expressed as:
463IdentityFilter <&&> someOtherFilter
466IdentityFilter <||> someOtherFilter
468-- Also there is a NotFilter, it only accepts a single filter, not a list.
470NotFilter someOtherFilter False
478-- And'ing two Identity
479let queryFilter = IdentityFilter <&&> IdentityFilter
481let search = mkSearch Nothing (Just queryFilter)
483reply <- searchByType testServer testIndex testMapping search
489Similar to boolean queries.
493-- Will return only items whose "user" field contains the term "bitemyapp"
494let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
496-- Will return only items whose "user" field does not contain the term "bitemyapp"
497let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
499-- The clause (query) should appear in the matching document.
500-- In a boolean query with no must clauses, one or more should
501-- clauses must match a document. The minimum number of should
502-- clauses to match can be set using the minimum_should_match parameter.
503let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
511-- Will filter for documents that have the field "user"
512let existsFilter = ExistsFilter (FieldName "user")
516**** Geo BoundingBox Filter
520-- topLeft and bottomRight
521let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
523let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False GeoFilterMemory
527**** Geo Distance Filter
531let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
533-- coefficient and units
534let distance = Distance 10.0 Miles
536-- GeoFilterType or NoOptimizeBbox
537let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
539-- SloppyArc is the usual/default optimization in Elasticsearch today
540-- but pre-1.0 versions will need to pick Arc or Plane.
542let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
546**** Geo Distance Range Filter
548Think of a donut and you won't be far off.
552let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
554let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
556let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
560**** Geo Polygon Filter
564-- I think I drew a square here.
565let points = [LatLon 40.0 (-70.00),
566 LatLon 40.0 (-72.00),
567 LatLon 41.0 (-70.00),
568 LatLon 41.0 (-72.00)]
570let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
574**** Document IDs filter
578-- takes a mapping name and a list of DocIds
579IdsFilter (MappingName "tweet") [DocId "1"]
589-- RangeFilter :: FieldName
590-- -> Either HalfRange Range
594let filter = RangeFilter (FieldName "age")
595 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
596 RangeExecutionIndex False
604let filter = RangeFilter (FieldName "age")
605 (Left (HalfRangeLt (LessThan 100000.0)))
606 RangeExecutionIndex False
622let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
623 AllRegexpFlags (CacheName "test") False (CacheKey "key")
626-- data RegexpFlags = AllRegexpFlags
628-- | SomeRegexpFlags (NonEmpty RegexpFlag) deriving (Eq, Show)
630-- data RegexpFlag = AnyString
635-- | Interval deriving (Eq, Show)
639* Possible future functionality
643Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
645** Function Score Query
647http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
649** Node discovery and failover
651Might require TCP support.
653** Support for TCP access to Elasticsearch
655Pretend to be a transport client?
657** Bulk cluster-join merge
659Might require making a lucene index on disk with the appropriate format.
663http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
667http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
669** Geohash cell filter
671http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
675http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
679http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
683http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
687http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
689** Script based sorting
691http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
693** Collapsing redundantly nested and/or structures
695The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
697** Runtime checking for cycles in data structures
699check for n > 1 occurrences in DFS:
701http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
703http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
707Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list