3#+CAPTION: Bloodhound (dog)
6* Elasticsearch client and query DSL for Haskell
10Search doesn't have to be hard. Let the dog do it.
14Bloodhound is alpha at the moment. The library works fine, but I don't want to mislead anyone into thinking the API is final or stable. I wouldn't call the library "complete" or representative of everything you can do in Elasticsearch but compared to clients in other languages, the usability is good.
24-- Formatted for use in ghci, so there are "let"s in front of the decls.
26-- if you see :{ and :}, they're so you can copy-paste
27-- the multi-line examples into your ghci REPL.
30import Database.Bloodhound
32import Data.Either (Either(..))
33import Data.Maybe (fromJust)
34import Data.Time.Calendar (Day(..))
35import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
36import Data.Text (Text)
37import GHC.Generics (Generic)
38import Network.HTTP.Conduit
39import qualified Network.HTTP.Types.Status as NHTS
41-- no trailing slashes in servers, library handles building the path.
42let testServer = (Server "http://localhost:9200")
43let testIndex = IndexName "twitter"
44let testMapping = MappingName "tweet"
46-- defaultIndexSettings is exported by Database.Bloodhound as well
47let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
49-- createIndex returns IO Reply
51-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
52response <- createIndex testServer defaultIndexSettings testIndex
61response <- deleteIndex testServer testIndex
63-- print response if it was a success
64Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
65 , responseVersion = HTTP/1.1
66 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
67 , ("Content-Length", "21")]
68 , responseBody = "{\"acknowledged\":true}"
69 , responseCookieJar = CJ {expose = []}
70 , responseClose' = ResponseClose}
72-- if the index to be deleted didn't exist anyway
73Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
74 , responseVersion = HTTP/1.1
75 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
76 , ("Content-Length","65")]
77 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
78 , responseCookieJar = CJ {expose = []}
79 , responseClose' = ResponseClose}
85**** Note, you *have* to do this if you expect to read what you just wrote
89resp <- refreshIndex testServer testIndex
91-- print resp on success
92Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
93 , responseVersion = HTTP/1.1
94 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
95 , ("Content-Length","50")]
96 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
97 , responseCookieJar = CJ {expose = []}
98 , responseClose' = ResponseClose}
108-- don't forget imports and the like at the top.
110data TweetMapping = TweetMapping deriving (Eq, Show)
112-- I know writing the JSON manually sucks.
113-- I don't have a proper data type for Mappings yet.
114-- Let me know if this is something you need.
117instance ToJSON TweetMapping where
118 toJSON TweetMapping =
120 object ["properties" .=
121 object ["location" .=
122 object ["type" .= ("geo_point" :: Text)]]]]
125resp <- createMapping testServer testIndex testMapping TweetMapping
133resp <- deleteMapping testServer testIndex testMapping
137** Document Operations
139*** Indexing Documents
143-- don't forget the imports and derive generic setting for ghci
144-- at the beginning of the examples.
147data Location = Location { lat :: Double
148 , lon :: Double } deriving (Eq, Generic, Show)
150data Tweet = Tweet { user :: Text
151 , postDate :: UTCTime
154 , location :: Location } deriving (Eq, Generic, Show)
156exampleTweet = Tweet { user = "bitemyapp"
158 (ModifiedJulianDay 55000)
159 (secondsToDiffTime 10)
160 , message = "Use haskell!"
162 , location = Location 40.12 (-71.34) }
164-- automagic (generic) derivation of instances because we're lazy.
166instance FromJSON Tweet
167instance ToJSON Location
168instance FromJSON Location
171-- Should be able to toJSON and encode the data structures like this:
172-- λ> toJSON $ Location 10.0 10.0
173-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
174-- λ> encode $ Location 10.0 10.0
175-- "{\"lat\":10,\"lon\":10}"
177resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
179-- print resp on success
180Response {responseStatus =
181 Status {statusCode = 200, statusMessage = "OK"}
182 , responseVersion = HTTP/1.1, responseHeaders =
183 [("Content-Type","application/json; charset=UTF-8"),
184 ("Content-Length","75")]
185 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
186 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
190*** Deleting Documents
194resp <- deleteDocument testServer testIndex testMapping (DocId "1")
202-- n.b., you'll need the earlier imports. responseBody is from http-conduit
204resp <- getDocument testServer testIndex testMapping (DocId "1")
206-- responseBody :: Response body -> body
207let body = responseBody resp
209-- you have two options, you use decode and just get Maybe (EsResult Tweet)
210-- or you can use eitherDecode and get Either String (EsResult Tweet)
212let maybeResult = decode body :: Maybe (EsResult Tweet)
213-- the explicit typing is so Aeson knows how to parse the JSON.
215-- use either if you want to know why something failed to parse.
216-- (string errors, sadly)
217let eitherResult = decode body :: Either String (EsResult Tweet)
219-- print eitherResult should look like:
220Right (EsResult {_index = "twitter"
225 , _source = Tweet {user = "bitemyapp"
226 , postDate = 2009-06-18 00:00:10 UTC
227 , message = "Use haskell!"
229 , location = Location {lat = 40.12, lon = -71.34}}})
231-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
233-- use the _source record accessor to get at your document
234λ> fmap _source result
235Right (Tweet {user = "bitemyapp"
236 , postDate = 2009-06-18 00:00:10 UTC
237 , message = "Use haskell!"
239 , location = Location {lat = 40.12, lon = -71.34}})
251-- exported by the Client module, just defaults some stuff.
252-- mkSearch :: Maybe Query -> Maybe Filter -> Search
253-- mkSearch query filter = Search query filter Nothing False 0 10
255let query = TermQuery (Term "user" "bitemyapp") Nothing
257-- AND'ing identity filter with itself and then tacking it onto a query
258-- search should be a null-operation. I include it for the sake of example.
259-- <||> (or/plus) should make it into a search that returns everything.
261let filter = IdentityFilter <&&> IdentityFilter
263-- constructing the search object the searchByIndex function dispatches on.
264let search = mkSearch (Just query) (Just filter)
266-- you can also searchByType and specify the mapping name.
267reply <- searchByIndex testServer testIndex search
269let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
271λ> fmap (hits . searchHits) result
272Right [Hit {hitIndex = IndexName "twitter"
273 , hitType = MappingName "tweet"
274 , hitDocId = DocId "1"
275 , hitScore = 0.30685282
276 , hitSource = Tweet {user = "bitemyapp"
277 , postDate = 2009-06-18 00:00:10 UTC
278 , message = "Use haskell!"
280 , location = Location {lat = 40.12, lon = -71.34}}}]
288let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
290-- mkSort is a shortcut function that takes a FieldName and a SortOrder
291-- to generate a vanilla DefaultSort.
292-- checkt the DefaultSort type for the full list of customizable options.
294-- From and size are integers for pagination.
296-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
298-- type Sort = [SortSpec]
299-- type TrackSortScores = Bool
303-- Search takes Maybe Query
309-- just add more sortspecs to the list if you want tie-breakers.
310let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
316**** And, Not, and Or filters
318Filters form a monoid and seminearring.
322instance Monoid Filter where
323 mempty = IdentityFilter
324 mappend a b = AndFilter [a, b] defaultCache
326instance Seminearring Filter where
327 a <||> b = OrFilter [a, b] defaultCache
329-- AndFilter and OrFilter take [Filter] as an argument.
331-- This will return anything, because IdentityFilter returns everything
332OrFilter [IdentityFilter, someOtherFilter] False
334-- This will return exactly what someOtherFilter returns
335AndFilter [IdentityFilter, someOtherFilter] False
337-- Thanks to the seminearring and monoid, the above can be expressed as:
340IdentityFilter <&&> someOtherFilter
343IdentityFilter <||> someOtherFilter
345-- Also there is a NotFilter, it only accepts a single filter, not a list.
347NotFilter someOtherFilter False
355-- And'ing two Identity
356let queryFilter = IdentityFilter <&&> IdentityFilter
358let search = mkSearch Nothing (Just queryFilter)
360reply <- searchByType testServer testIndex testMapping search
366Similar to boolean queries.
370-- Will return only items whose "user" field contains the term "bitemyapp"
371let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
373-- Will return only items whose "user" field does not contain the term "bitemyapp"
374let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
376-- The clause (query) should appear in the matching document.
377-- In a boolean query with no must clauses, one or more should
378-- clauses must match a document. The minimum number of should
379-- clauses to match can be set using the minimum_should_match parameter.
380let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
388-- Will filter for documents that have the field "user"
389let existsFilter = ExistsFilter (FieldName "user")
393**** Geo BoundingBox Filter
397-- topLeft and bottomRight
398let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
400let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False
402-- second argument is GeoFilterType, memory or indexed.
403let geoFilter = GeoBoundingBoxFilter constraint GeoFilterMemory
407**** Geo Distance Filter
411let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
413-- coefficient and units
414let distance = Distance 10.0 Miles
416-- GeoFilterType or NoOptimizeBbox
417let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
419-- SloppyArc is the usual/default optimization in Elasticsearch today
420-- but pre-1.0 versions will need to pick Arc or Plane.
422let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
426**** Geo Distance Range Filter
428Think of a donut and you won't be far off.
432let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
434let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
436let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
440**** Geo Polygon Filter
444-- I think I drew a square here.
445let points = [LatLon 40.0 (-70.00),
446 LatLon 40.0 (-72.00),
447 LatLon 41.0 (-70.00),
448 LatLon 41.0 (-72.00)]
450let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
454**** Document IDs filter
458-- takes a mapping name and a list of DocIds
459IdsFilter (MappingName "tweet") [DocId "1"]
469-- RangeFilter :: FieldName
470-- -> Either HalfRange Range
474let filter = RangeFilter (FieldName "age")
475 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
476 RangeExecutionIndex False
484let filter = RangeFilter (FieldName "age")
485 (Left (HalfRangeLt (LessThan 100000.0)))
486 RangeExecutionIndex False
502let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
503 RegexpAll (CacheName "test") False (CacheKey "key")
505-- RegexpFlags can be a combination of RegexpAll, Complement,
506-- Interval, Intersection, AnyString, and a combination of two options thereof.
510* Possible future functionality
514Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
516** Function Score Query
518http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
520** Node discovery and failover
522Might require TCP support.
524** Support for TCP access to Elasticsearch
526Pretend to be a transport client?
528** Bulk cluster-join merge
530Might require making a lucene index on disk with the appropriate format.
534http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
538http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
540** Geohash cell filter
542http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
546http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
550http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
554http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
558http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
560** Script based sorting
562http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
564** Collapsing redundantly nested and/or structures
566The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
568** Runtime checking for cycles in data structures
570check for n > 1 occurrences in DFS:
572http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
574http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
578Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list