3#+CAPTION: Bloodhound (dog)
8[[https://travis-ci.org/bitemyapp/bloodhound][https://travis-ci.org/bitemyapp/bloodhound.svg]]
10* Elasticsearch client and query DSL for Haskell
14Search doesn't have to be hard. Let the dog do it.
16** Version compatibility
18Elasticsearch >= 1.0 is recommended. Bloodhound mostly works with 0.9.x, but I don't recommend it if you expect everything to work. As of Bloodhound 0.3 all >=1.0 versions of Elasticsearch work.
20Current versions we test against are 1.0.3, 1.1.2, 1.2.3, and 1.3.2. We also check that GHC 7.6 and 7.8 both build and pass tests. See our [TravisCI](https://travis-ci.org/bitemyapp/bloodhound) to learn more.
24Bloodhound is beta at the moment. We've got a solid library tested across multiple ES versions, but coverage of Elasticsearch functionality isn't 100% yet.
26* Hackage page and Haddock documentation
28http://hackage.haskell.org/package/bloodhound
38-- Formatted for use in ghci, so there are "let"s in front of the decls.
40-- if you see :{ and :}, they're so you can copy-paste
41-- the multi-line examples into your ghci REPL.
44import Database.Bloodhound
46import Data.Either (Either(..))
47import Data.Maybe (fromJust)
48import Data.Time.Calendar (Day(..))
49import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
50import Data.Text (Text)
51import GHC.Generics (Generic)
52import Network.HTTP.Conduit
53import qualified Network.HTTP.Types.Status as NHTS
55-- no trailing slashes in servers, library handles building the path.
56let testServer = (Server "http://localhost:9200")
57let testIndex = IndexName "twitter"
58let testMapping = MappingName "tweet"
60-- defaultIndexSettings is exported by Database.Bloodhound as well
61let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
63-- createIndex returns IO Reply
65-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
66response <- createIndex testServer defaultIndexSettings testIndex
77response <- deleteIndex testServer testIndex
85-- print response if it was a success
86Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
87 , responseVersion = HTTP/1.1
88 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
89 , ("Content-Length", "21")]
90 , responseBody = "{\"acknowledged\":true}"
91 , responseCookieJar = CJ {expose = []}
92 , responseClose' = ResponseClose}
94-- if the index to be deleted didn't exist anyway
95Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
96 , responseVersion = HTTP/1.1
97 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
98 , ("Content-Length","65")]
99 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
100 , responseCookieJar = CJ {expose = []}
101 , responseClose' = ResponseClose}
107**** Note, you *have* to do this if you expect to read what you just wrote
111resp <- refreshIndex testServer testIndex
119-- print resp on success
120Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
121 , responseVersion = HTTP/1.1
122 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
123 , ("Content-Length","50")]
124 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
125 , responseCookieJar = CJ {expose = []}
126 , responseClose' = ResponseClose}
136-- don't forget imports and the like at the top.
138data TweetMapping = TweetMapping deriving (Eq, Show)
140-- I know writing the JSON manually sucks.
141-- I don't have a proper data type for Mappings yet.
142-- Let me know if this is something you need.
145instance ToJSON TweetMapping where
146 toJSON TweetMapping =
148 object ["properties" .=
149 object ["location" .=
150 object ["type" .= ("geo_point" :: Text)]]]]
153resp <- putMapping testServer testIndex testMapping TweetMapping
161resp <- deleteMapping testServer testIndex testMapping
165** Document Operations
167*** Indexing Documents
171-- don't forget the imports and derive generic setting for ghci
172-- at the beginning of the examples.
175data Location = Location { lat :: Double
176 , lon :: Double } deriving (Eq, Generic, Show)
178data Tweet = Tweet { user :: Text
179 , postDate :: UTCTime
182 , location :: Location } deriving (Eq, Generic, Show)
184exampleTweet = Tweet { user = "bitemyapp"
186 (ModifiedJulianDay 55000)
187 (secondsToDiffTime 10)
188 , message = "Use haskell!"
190 , location = Location 40.12 (-71.34) }
192-- automagic (generic) derivation of instances because we're lazy.
194instance FromJSON Tweet
195instance ToJSON Location
196instance FromJSON Location
199-- Should be able to toJSON and encode the data structures like this:
200-- λ> toJSON $ Location 10.0 10.0
201-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
202-- λ> encode $ Location 10.0 10.0
203-- "{\"lat\":10,\"lon\":10}"
205resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
213Response {responseStatus =
214 Status {statusCode = 200, statusMessage = "OK"}
215 , responseVersion = HTTP/1.1, responseHeaders =
216 [("Content-Type","application/json; charset=UTF-8"),
217 ("Content-Length","75")]
218 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
219 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
223*** Deleting Documents
227resp <- deleteDocument testServer testIndex testMapping (DocId "1")
235-- n.b., you'll need the earlier imports. responseBody is from http-conduit
237resp <- getDocument testServer testIndex testMapping (DocId "1")
239-- responseBody :: Response body -> body
240let body = responseBody resp
242-- you have two options, you use decode and just get Maybe (EsResult Tweet)
243-- or you can use eitherDecode and get Either String (EsResult Tweet)
245let maybeResult = decode body :: Maybe (EsResult Tweet)
246-- the explicit typing is so Aeson knows how to parse the JSON.
248-- use either if you want to know why something failed to parse.
249-- (string errors, sadly)
250let eitherResult = decode body :: Either String (EsResult Tweet)
252-- print eitherResult should look like:
253Right (EsResult {_index = "twitter"
258 , _source = Tweet {user = "bitemyapp"
259 , postDate = 2009-06-18 00:00:10 UTC
260 , message = "Use haskell!"
262 , location = Location {lat = 40.12, lon = -71.34}}})
264-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
266-- use the _source record accessor to get at your document
267λ> fmap _source result
268Right (Tweet {user = "bitemyapp"
269 , postDate = 2009-06-18 00:00:10 UTC
270 , message = "Use haskell!"
272 , location = Location {lat = 40.12, lon = -71.34}})
284-- exported by the Client module, just defaults some stuff.
285-- mkSearch :: Maybe Query -> Maybe Filter -> Search
286-- mkSearch query filter = Search query filter Nothing False 0 10
288let query = TermQuery (Term "user" "bitemyapp") Nothing
290-- AND'ing identity filter with itself and then tacking it onto a query
291-- search should be a null-operation. I include it for the sake of example.
292-- <||> (or/plus) should make it into a search that returns everything.
294let filter = IdentityFilter <&&> IdentityFilter
296-- constructing the search object the searchByIndex function dispatches on.
297let search = mkSearch (Just query) (Just filter)
299-- you can also searchByType and specify the mapping name.
300reply <- searchByIndex testServer testIndex search
302let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
304λ> fmap (hits . searchHits) result
305Right [Hit {hitIndex = IndexName "twitter"
306 , hitType = MappingName "tweet"
307 , hitDocId = DocId "1"
308 , hitScore = 0.30685282
309 , hitSource = Tweet {user = "bitemyapp"
310 , postDate = 2009-06-18 00:00:10 UTC
311 , message = "Use haskell!"
313 , location = Location {lat = 40.12, lon = -71.34}}}]
321let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
322let search = mkSearch (Just query) Nothing
327**** Multi-Match Query
331let fields = [FieldName "user", FieldName "message"]
332let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
333let search = mkSearch (Just query) Nothing
341let innerQuery = QueryMatchQuery $
342 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
343let query = QueryBoolQuery $
344 mkBoolQuery (Just innerQuery) Nothing Nothing
345let search = mkSearch (Just query) Nothing
353let posQuery = QueryMatchQuery $
354 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
355let negQuery = QueryMatchQuery $
356 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
357let query = QueryBoostingQuery $
358 BoostingQuery posQuery negQuery (Boost 0.2)
362**** Rest of the query/filter types
364Just follow the pattern you've seen here and check the Hackage API documentation.
370let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
372-- mkSort is a shortcut function that takes a FieldName and a SortOrder
373-- to generate a vanilla DefaultSort.
374-- checkt the DefaultSort type for the full list of customizable options.
376-- From and size are integers for pagination.
378-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
380-- type Sort = [SortSpec]
381-- type TrackSortScores = Bool
385-- Search takes Maybe Query
391-- just add more sortspecs to the list if you want tie-breakers.
392let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
398**** And, Not, and Or filters
400Filters form a monoid and seminearring.
404instance Monoid Filter where
405 mempty = IdentityFilter
406 mappend a b = AndFilter [a, b] defaultCache
408instance Seminearring Filter where
409 a <||> b = OrFilter [a, b] defaultCache
411-- AndFilter and OrFilter take [Filter] as an argument.
413-- This will return anything, because IdentityFilter returns everything
414OrFilter [IdentityFilter, someOtherFilter] False
416-- This will return exactly what someOtherFilter returns
417AndFilter [IdentityFilter, someOtherFilter] False
419-- Thanks to the seminearring and monoid, the above can be expressed as:
422IdentityFilter <&&> someOtherFilter
425IdentityFilter <||> someOtherFilter
427-- Also there is a NotFilter, it only accepts a single filter, not a list.
429NotFilter someOtherFilter False
437-- And'ing two Identity
438let queryFilter = IdentityFilter <&&> IdentityFilter
440let search = mkSearch Nothing (Just queryFilter)
442reply <- searchByType testServer testIndex testMapping search
448Similar to boolean queries.
452-- Will return only items whose "user" field contains the term "bitemyapp"
453let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
455-- Will return only items whose "user" field does not contain the term "bitemyapp"
456let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
458-- The clause (query) should appear in the matching document.
459-- In a boolean query with no must clauses, one or more should
460-- clauses must match a document. The minimum number of should
461-- clauses to match can be set using the minimum_should_match parameter.
462let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
470-- Will filter for documents that have the field "user"
471let existsFilter = ExistsFilter (FieldName "user")
475**** Geo BoundingBox Filter
479-- topLeft and bottomRight
480let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
482let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False GeoFilterMemory
486**** Geo Distance Filter
490let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
492-- coefficient and units
493let distance = Distance 10.0 Miles
495-- GeoFilterType or NoOptimizeBbox
496let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
498-- SloppyArc is the usual/default optimization in Elasticsearch today
499-- but pre-1.0 versions will need to pick Arc or Plane.
501let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
505**** Geo Distance Range Filter
507Think of a donut and you won't be far off.
511let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
513let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
515let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
519**** Geo Polygon Filter
523-- I think I drew a square here.
524let points = [LatLon 40.0 (-70.00),
525 LatLon 40.0 (-72.00),
526 LatLon 41.0 (-70.00),
527 LatLon 41.0 (-72.00)]
529let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
533**** Document IDs filter
537-- takes a mapping name and a list of DocIds
538IdsFilter (MappingName "tweet") [DocId "1"]
548-- RangeFilter :: FieldName
549-- -> Either HalfRange Range
553let filter = RangeFilter (FieldName "age")
554 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
555 RangeExecutionIndex False
563let filter = RangeFilter (FieldName "age")
564 (Left (HalfRangeLt (LessThan 100000.0)))
565 RangeExecutionIndex False
581let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
582 AllRegexpFlags (CacheName "test") False (CacheKey "key")
585-- data RegexpFlags = AllRegexpFlags
587-- | SomeRegexpFlags (NonEmpty RegexpFlag) deriving (Eq, Show)
589-- data RegexpFlag = AnyString
594-- | Interval deriving (Eq, Show)
598* Possible future functionality
602Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
604** Function Score Query
606http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
608** Node discovery and failover
610Might require TCP support.
612** Support for TCP access to Elasticsearch
614Pretend to be a transport client?
616** Bulk cluster-join merge
618Might require making a lucene index on disk with the appropriate format.
622http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
626http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
628** Geohash cell filter
630http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
634http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
638http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
642http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
646http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
648** Script based sorting
650http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
652** Collapsing redundantly nested and/or structures
654The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
656** Runtime checking for cycles in data structures
658check for n > 1 occurrences in DFS:
660http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
662http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
666Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list