graphql-engine/server/lib/ekg-prometheus/Tutorial.md

NOTE: This tutorial is not being maintained. It needs to be rewritten.

Tutorial

This document introduces the ekg-prometheus Prometheus client library, and illustrates how to use the library to instrument your programs with Prometheus metrics. If you are new to the library, read this document first. If you have used the ekg-core library, on which ekg-prometheus is based, you should still read this document first. For a more complete API reference, see the Haddocks of the System.Metrics.Prometheus module.

This document is a literate Haskell program:

-- Note: The code in this tutorial is not being maintained.
main :: IO ()
main = pure ()

-- {-# LANGUAGE DataKinds #-}
-- {-# LANGUAGE DeriveGeneric #-}
-- {-# LANGUAGE GADTs #-}
-- {-# LANGUAGE KindSignatures #-}
-- {-# LANGUAGE OverloadedStrings #-}
-- {-# LANGUAGE TypeApplications #-}
-- 
-- module Main where
-- 
-- import Control.Exception (assert)
-- import qualified Data.HashMap.Strict as HM
-- import qualified Data.Map.Strict as M
-- import qualified Data.Text as T
-- import Data.Kind (Type)
-- import GHC.Generics (Generic)
-- import GHC.Stats (RTSStats (..), getRTSStats)
-- import GHC.TypeLits (Symbol)
-- 
-- -- This package's modules
-- import System.Metrics.Prometheus
-- import qualified System.Metrics.Prometheus.Counter as Counter
-- import qualified System.Metrics.Prometheus.Gauge as Gauge

Although you will need to use some type-level features of Haskell when using the ekg-prometheus API, you will not need a solid understanding of type-level programming. You can use ekg-prometheus proficiently just by copying the examples presented in this tutorial.

For those who have used the original ekg-core library, Hasura's fork adds the following features:

• dimensional/tagged metrics (Prometheus labels), and
• dynamic metrics (the ability to deregister and reregister metrics).

Overview

Metrics are used to monitor program behavior and performance. All metrics have:

• a name,
• a set of labels (possibly empty), and
• a way to get the metric's current value.

ekg-prometheus provides a way to register metrics in a global "metric store". The store can then be used to get a snapshot of all metrics. The store also serves as a central place to keep track of all the program's metrics, both user and library defined.

This tutorial will show you how to:

• specify metrics,
• register and sample metrics,
• add labels to metrics,
• deregister metrics,
• use pre-defined metrics, and
• sample a subset of metrics atomically.

Specifying metrics

Before you can register metrics to a metric store, you must first specify which metrics may be registered to that store. ekg-prometheus will statically ensure that your specifications are respected.

Your metrics specification must be given as a generalized algebraic data type (GADT) with a specific kind signature. Here is an example GADT that specifies two metrics:

-- data AppMetrics1
--   :: Symbol -- ^ Metric name
--   -> Symbol -- ^ Metric documentation
--   -> MetricType -- ^ e.g. Counter, Gauge
--   -> Type -- ^ Label set structure
--   -> Type
--   where
--   Requests :: AppMetrics1 "app_requests" "" 'CounterType ()
--   Connections :: AppMetrics1 "app_connections" "" 'GaugeType ()

The AppMetrics1 GADT has two constructors, Requests and Connections, each of which correspond to a metric. The type parameters of each constructor determine the name, type, and "label structure" of their corresponding metric. For example, the Requests constructor specifies a metric with:

• name "app_requests", and
• type counter, and
• labels disabled.

Tutorial note: We have glossed over labels for now, but will introduce them properly later.

Registering and sampling metrics

Now that you have created a metrics specification, you can use it to annotate a metric store and start registering and collecting metrics.

Here is an example program that uses the above specification:

-- app1 :: IO ()
-- app1 = do
--   -- Create a mutable reference to a metric store.
--   store <- newStore @AppMetrics1 -- (1)
-- 
--   -- Initialize mutable references to metrics.
--   requestsCounter <- Counter.new
--   connectionsGauge <- Gauge.new
-- 
--   -- Register the metrics to the metric store.
--   _ <- register store $ -- (2)
--     registerCounter Requests () (Counter.read requestsCounter) <>
--     registerGauge Connections () (Gauge.read connectionsGauge)
-- 
--   -- Update the values of the metrics.
--   Counter.inc requestsCounter
--   Gauge.set connectionsGauge 99
-- 
--   -- Get the current values of all the metrics in the store.
--   sample <- sampleAll store -- (3)
-- 
--   -- Verify the sample, just for this tutorial.
--   let expectedSample = M.fromList
--         [ ("app_requests", ("", M.singleton HM.empty (Counter 1)))
--         , ("app_connections", ("", M.singleton HM.empty (Gauge 99)))
--         ]
--   assert (sample == expectedSample) $ pure ()

1. Metric store references are parameterized by a metrics specification. In this case, we have used -XTypeApplications to explicitly name the intended metrics specification, even though GHC could infer the metrics specification itself.

2. The register IO action atomically applies a sequence of "registrations" to a metric store. Individual registrations are created by functions like registerCounter and registerGauge, and can be combined into a sequence of registrations by their Semigroup operation <>.

   The registerCounter function takes as its first argument a constructor of a metrics specification GADT. This constructor must have metric type 'CounterType. Its second parameter specifies the set of "labels" to attach to the metric -- for now, labels have been disallowed. Its third parameter specifies the IO action that the store should use to sample the current value of the metric.

   The registerGauge function is the analogue of registerCounter for the gauge metric type.

3. The sampleAll function iterates through all of the metrics registered to the store, runs their sampling actions in turn, and collects the results. Note that sampling is not atomic: While each metric will be retrieved atomically, the sample is not an atomic snapshot of the system as a whole. (For more information, see sampling metrics atomically)

Adding labels to metrics

ekg-prometheus has a multi-dimensional data model, like Prometheus. In this data model, metrics may be annotated by a labels set, which is a set of key-value pairs called labels. Labels are useful for convenient filtering and aggregation of metric data. In ekg-prometheus, metrics are identified by both their name and their label set, so metrics with the same name but different label sets are distinct and independent metrics. When working with labelled metrics, the constructors of a metrics specification GADT corrrespond to classes of metrics that share the same name.

ekg-prometheus also has support for structuring the representation of your labels. A label set can be represented by a value of any type, as long as the type is associated with a function that "renders" the value into a label set. More specifically, a label set can be represented by a value of any type that is an instance of the ToLabels typeclass, which has a single function toLabels :: ToLabels a => a -> HashMap Text Text.

Here is an example metrics specification that defines some labelled metrics:

-- data AppMetrics2
--   :: Symbol
--   -> Symbol
--   -> MetricType
--   -> Type -- ^ Label set structure
--   -> Type
--   where
--   -- (1)
--   HTTPRequests ::
--     AppMetrics2 "requests" "" 'CounterType EndpointLabels
--   DBConnections ::
--     AppMetrics2 "total_connections" "" 'GaugeType DataSourceLabels
-- 
-- -- (2)
-- newtype EndpointLabels = EndpointLabels { endpoint :: T.Text }
-- 
-- instance ToLabels EndpointLabels where
--   toLabels (EndpointLabels endpoint') = HM.singleton "endpoint" endpoint'
-- 
-- -- 3
-- data DataSourceLabels = DataSourceLabels
--   { source_name :: T.Text
--   , conn_info :: T.Text
--   } deriving (Generic)
-- instance ToLabels DataSourceLabels

1. The third type parameter of the constructors is used to specify label set structure.

   In this example, the types provided for the label set structure parameter are two user-defined types, EndpointLabels and DataSourceLabels.

2. Here, the ToLabels instance of EndpointLabels has been specified by hand.

3. Here, the ToLabels instance of DataSourceLabels has been specified using GHC.Generics.

   A ToLabels instance may be derived via GHC.Generics for any record that exclusively has fields of type Text. The record field names are used as the label keys.

Here is an example program using this metrics specification:

-- app2 :: IO ()
-- app2 = do
--   store <- newStore @AppMetrics2
-- 
--   harpsichordRequests <- Counter.new
--   tablaRequests <- Counter.new
--   dbConnections <- Gauge.new
-- 
--   _ <- register store $ mconcat
--     [ registerCounter HTTPRequests (EndpointLabels "dev/harpsichord") (Counter.read harpsichordRequests)
--     , registerCounter HTTPRequests (EndpointLabels "dev/tabla") (Counter.read tablaRequests)
--     , let labels = DataSourceLabels
--             { source_name = "myDB"
--             , conn_info = "localhost:5432" }
--       in  registerGauge DBConnections labels (Gauge.read dbConnections)
--     ]
-- 
--   Counter.inc tablaRequests
--   Gauge.set dbConnections 99
-- 
--   sample <- sampleAll store
-- 
--   let expectedSample = M.fromList
--         [ ( "requests"
--           , ( ""
--             , M.fromList
--                 [ (HM.singleton "endpoint" "dev/harpsichord", Counter 0)
--                 , (HM.singleton "endpoint" "dev/tabla", Counter 1)
--                 ]
--             )
--           )
--         , ( "total_connections"
--           , ( ""
--             , M.singleton
--                 ( HM.fromList
--                   [ ("source_name", "myDB")
--                   , ("conn_info", "localhost:5432")
--                   ]
--                 )
--                 (Gauge 99)
--             )
--           )
--         ]
--   assert (sample == expectedSample) $ pure ()

Reregistering and deregistering metrics

Metrics you register to a metric store need not be permanent; metrics can be replaced (reregistered) or removed (deregistered).

Reregistering metrics in ekg-prometheus is implicit. If you try to register a metric at a (name, label set) pair that is already in use by an existing metric, the existing metric will be deregistered and replaced with the new metric.

Deregistering metrics in ekg-prometheus is explicit, and is done using deregistration handles. When you register a set of metrics with register, register will return an IO action (the deregistration handle) that can be used to specifically deregister the newly registered metrics. This action is specific in the following sense: if a deregistration handle targets a metric, and that metric is replaced by a new metric, the new metric will not be deregistered if the handle is used.

Here is an example program that illustrates the reregistration and deregistration of metrics:

-- app3 :: IO ()
-- app3 = do
--   store <- newStore @AppMetrics1 -- reusing a previous specification
-- 
--   requestsCounter <- Counter.new
--   connectionsGauge <- Gauge.new
-- 
--   -- Register the metrics, retaining the deregistration handle. -- (1)
--   deregistrationHandle <- register store $
--     registerCounter Requests () (Counter.read requestsCounter) <>
--     registerGauge Connections () (Gauge.read connectionsGauge)
-- 
--   Counter.inc requestsCounter
--   Gauge.set connectionsGauge 99
-- 
--   sample1 <- sampleAll store
--   let expectedSample1 = M.fromList
--         [ ("app_requests", ("", M.singleton HM.empty (Counter 1)))
--         , ("app_connections", ("", M.singleton HM.empty (Gauge 99)))
--         ]
--   assert (sample1 == expectedSample1) $ pure ()
-- 
--   -- Replace (reregister) the connections gauge metric with a new one.
--   replacementConnectionsGauge <- Gauge.new
--   Gauge.set replacementConnectionsGauge 5
--   _ <- register store $
--     registerGauge Connections () (Gauge.read replacementConnectionsGauge)
-- 
--   sample2 <- sampleAll store
--   let expectedSample2 = M.fromList
--         [ ("app_requests", ("", M.singleton HM.empty (Counter 1)))
--         , ("app_connections", ("", M.singleton HM.empty (Gauge 5)))
--         ]
--   assert (sample2 == expectedSample2) $ pure ()
-- 
--   -- Use the deregistration handle to deregister the original metrics.
--   deregistrationHandle -- (2)
-- 
--   sample3 <- sampleAll store
--   let expectedSample3 =
--         M.singleton "app_connections" $
--           ("", M.singleton HM.empty (Gauge 5))
--   assert (sample3 == expectedSample3) $ pure ()

1. Deregistration handles were present in in all previous examples, but we ignored them for simplicity.

2. The deregistration handle removes all metrics registered by the initial call to register. In particular, this does not include the reregistered gauge.

Using pre-defined sets of metrics

Other libraries can define sets of metrics that you can register to your metric store. For example, the ekg-prometheus library defines metrics for the runtime system metrics exposed by GHC.Stats -- see registerGcMetrics. Libraries that define metrics must also define their own metrics specifications, which you will need to include in your own metrics specification in order to use their metrics.

Here is an example program which includes the GcMetrics metrics specification (used by registerGcMetrics) as a part of another metrics specification:

-- data AppMetrics4 :: Symbol -> Symbol -> MetricType -> Type -> Type where
--   -- (1)
--   GcSubset ::
--     GcMetrics name help metricType labels ->
--     AppMetrics4 name help metricType labels
-- 
-- app4 :: IO ()
-- app4 = do
--   store <- newStore @AppMetrics4
--   -- (2)
--   _ <- register (subset GcSubset store) registerGcMetrics
--   pure ()

1. We define a constructor, GcSubset, that takes any metric class from GcMetrics and makes it a metric class of AppMetrics4.

   Metric classes with the same type parameters (name, metric type, and label structure) are treated in the same way by all functions of ekg-prometheus, so it is enough for our constructor to "forward" the type parameters.

2. In order use registerGcMetrics with our metric store, we must use the subset function to create a new reference to our metric store restricted to the GcMetrics metrics specification that registerGcMetrics expects.

Sampling groups of metrics atomically

ekg-prometheus provides a way to obtain atomic snapshots of a group of metrics. This can be useful if

• you need a consistent view of several metrics, or
• sampling the metrics together is more efficient.

For example, sampling GC statistics needs to be done atomically or a GC might strike in the middle of sampling, rendering the values incoherent. Sampling GC statistics is also more efficient if done in "bulk", as the run-time system provides a function to sample all GC statistics at once.

The usual metric samples obtained through the sampleAll function are generally not atomic snapshots of their metrics. In general, because metric sampling actions can be arbitrary IO actions, ekg-prometheus has no way to ensure that independent metrics are sampled atomically.

However, a group of metrics can be sampled atomically if

• their values are all derived from a single shared value, via pure functions, and
• the IO action that computes the shared value does so atomically (e.g. if the shared value is a record, the action needs to compute its fields atomically).

To register an atomically-sampled group of metrics, use the registerGroup function and the SamplingGroup type. Here is an example program that does this:

-- -- (1)
-- data GcMetrics' :: Symbol -> Symbol -> MetricType -> Type -> Type where
--   Gcs' :: GcMetrics' "rts_gcs" "" 'CounterType ()
--   MaxLiveBytes' :: GcMetrics' "rts_max_live_bytes" "" 'GaugeType ()
-- 
-- app5 :: IO ()
-- app5 = do
--   store <- newStore @GcMetrics'
-- 
--   -- (2)
--   let samplingGroup =
--         SamplingGroup
--           :> (Gcs', (), fromIntegral . gcs)
--           :> (MaxLiveBytes', (), fromIntegral . max_live_bytes)
-- 
--   _ <- register store $
--         registerGroup samplingGroup getRTSStats -- (3)
--   pure ()

1. We replicate part of the GcMetrics metrics specification from ekg-prometheus.

2. We create a sampling group of two of the runtime system metrics.

   Each metric is represented by:

   • a metric class,
   • a label set, and
   • a pure function that computes the metric's value from a single value that is shared with all metrics of the sampling group.

3. We use the registerGroup function to pair our sampling group with an IO action, getRTSStats, that produces the shared value.

Conclusion

This tutorial introduced and demonstrated the core features of the ekg-prometheus library:

• specifying metrics,
• registering and sampling metrics,
• labelling metrics,
• deregistering metrics,
• using pre-defined metrics, and
• sampling a subset of metrics atomically.

Additional features and details can be found in the following documents:

• the Haddocks for the System.Metrics.Prometheus module
• the Appendix section below.

Appendix

This section contains extra material that is not needed to use the ekg-prometheus library, but may be useful. This section assumes an understanding of the material covered in the tutorial.

Simulating static metrics

You can register metrics to a metric store so that they cannot be removed or modified. Here is an example program that does this.

-- -- (1)
-- data AppMetrics6 :: Symbol -> Symbol -> MetricType -> Type -> Type where
--   DynamicSubset ::
--     DynamicMetrics name help metricType labels ->
--     AppMetrics6 name help metricType labels
--   StaticSubset ::
--     StaticMetrics name help metricType labels ->
--     AppMetrics6 name help metricType labels
-- 
-- data StaticMetrics :: Symbol -> Symbol -> MetricType -> Type -> Type where
--   MyStaticMetric :: StaticMetrics "my_static_metric" "" 'CounterType ()
-- 
-- data DynamicMetrics :: Symbol -> Symbol -> MetricType -> Type -> Type where
--   MyDynamicMetric :: DynamicMetrics "my_dynamic_metric" "" 'CounterType ()
-- 
-- app6 :: IO ()
-- app6 = do
--   (_store, _staticMetrics) <- do
--     store <- newStore @AppMetrics6
--     -- (2)
--     let staticRef = subset StaticSubset store
--         dynamicRef = subset DynamicSubset store
--     staticMetrics <- registerStaticMetrics staticRef
--     pure (dynamicRef, staticMetrics)
-- 
--   -- (3)
--   pure ()
-- 
-- registerStaticMetrics :: Store StaticMetrics -> IO Counter.Counter
-- registerStaticMetrics store = do
--   counter <- Counter.new
--   _ <- register store $
--         registerCounter MyStaticMetric () (Counter.read counter)
--   pure counter

1. We divide our metrics specification into two subsets: one for static metrics that should not be removed or modified after being registered, and the other for dynamic metrics that may need to be removed or modified.

2. We use the subset function twice to create restricted references to the metric store. The first reference is scoped to the static subset, which we use to register the static metrics. The second reference is scoped to the dynamic subset, and is the only reference to the metric store that we expose.

3. At this point, the only reference to the store is scoped to the subset of dynamic metrics. There is no way to register or deregister metrics from the static subset, making those metrics effectively immutable.

Tutorial verification

This tutorial is compiled and run as a test using the markdown-unlit package.

-- main :: IO ()
-- main = do
--   app1
--   app2
--   app3
--   app4
--   app5
--   app6