mirror of
https://github.com/hasura/graphql-engine.git
synced 2024-12-15 09:22:43 +03:00
docs: add prometheus grafana dashboard
[DOCS-1025]: https://hasurahq.atlassian.net/browse/DOCS-1025?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ PR-URL: https://github.com/hasura/graphql-engine-mono/pull/9377 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> GitOrigin-RevId: d60a53c5a0937f1ea2eb087a84df59e4121e61c8
This commit is contained in:
parent
ee255fec61
commit
d711392fb1
docs
docs
static/img/enterprise/prometheus
@ -83,8 +83,8 @@ to be exported to your observability platform using the appropriate log drivers.
|
|||||||
#### Metrics via Prometheus integration
|
#### Metrics via Prometheus integration
|
||||||
|
|
||||||
You can export metrics of your Hasura Cloud project to Prometheus. You can configure this on the `Integrations` tab on
|
You can export metrics of your Hasura Cloud project to Prometheus. You can configure this on the `Integrations` tab on
|
||||||
the project's settings page. You can find more information on this [here](/enterprise/metrics.mdx). This page also
|
the project's settings page. You can find more information on this
|
||||||
provides information about different metrics exported from Hasura to Prometheus.
|
[here](/observability/prometheus/cloud-integration.mdx).
|
||||||
|
|
||||||
## Database observability
|
## Database observability
|
||||||
|
|
||||||
@ -107,8 +107,8 @@ be implemented:
|
|||||||
|
|
||||||
[Query Tags](/observability/query-tags.mdx) are SQL comments that consist of `key=value` pairs that are appended to
|
[Query Tags](/observability/query-tags.mdx) are SQL comments that consist of `key=value` pairs that are appended to
|
||||||
generated SQL statements. When you issue a query or mutation with query tags, the generated SQL has some extra
|
generated SQL statements. When you issue a query or mutation with query tags, the generated SQL has some extra
|
||||||
information. Database analytics tools can use that information (metadata) in these comments to analyze DB load
|
information. Database analytics tools can use that information (metadata) in these comments to analyze DB load and track
|
||||||
and track or monitor performance.
|
or monitor performance.
|
||||||
|
|
||||||
### Using Query Tags and pganalyze
|
### Using Query Tags and pganalyze
|
||||||
|
|
||||||
|
@ -194,8 +194,8 @@ Whether or not to send the request body (graphql request/variables) to the auth
|
|||||||
|
|
||||||
Stringify certain
|
Stringify certain
|
||||||
[BigQuery numeric types](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types),
|
[BigQuery numeric types](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types),
|
||||||
specifically `bignumeric`, `float64`, `int64`, `numeric` and aliases thereof, as they don't fit into the `IEnterprise EditionE 754` spec
|
specifically `bignumeric`, `float64`, `int64`, `numeric` and aliases thereof, as they don't fit into the
|
||||||
for JSON encoding-decoding.
|
`IEnterprise EditionE 754` spec for JSON encoding-decoding.
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------------------- | ---------------------------------------------- |
|
| ------------------- | ---------------------------------------------- |
|
||||||
@ -211,14 +211,14 @@ for JSON encoding-decoding.
|
|||||||
When metadata changes, close all WebSocket connections (with error code `1012`). This is useful when you want to ensure
|
When metadata changes, close all WebSocket connections (with error code `1012`). This is useful when you want to ensure
|
||||||
that all clients reconnect to the latest metadata.
|
that all clients reconnect to the latest metadata.
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------------------- | ----------------------------------------------------- |
|
| ------------------- | ---------------------------------------------------- |
|
||||||
| **Flag** | `--disable-close-websockets-on-metadata-change` |
|
| **Flag** | `--disable-close-websockets-on-metadata-change` |
|
||||||
| **Env var** | `HASURA_GRAPHQL_CLOSE_WEBSOCKETS_ON_METADATA_CHANGE` |
|
| **Env var** | `HASURA_GRAPHQL_CLOSE_WEBSOCKETS_ON_METADATA_CHANGE` |
|
||||||
| **Accepted values** | Boolean |
|
| **Accepted values** | Boolean |
|
||||||
| **Options** | `true` or `false` |
|
| **Options** | `true` or `false` |
|
||||||
| **Default** | `true` |
|
| **Default** | `true` |
|
||||||
| **Supported in** | CE, Enterprise Edition, Cloud |
|
| **Supported in** | CE, Enterprise Edition, Cloud |
|
||||||
|
|
||||||
### Connections per Read-Replica
|
### Connections per Read-Replica
|
||||||
|
|
||||||
@ -355,21 +355,21 @@ Enable the Hasura Console (served by the server on `/` and `/console`).
|
|||||||
|
|
||||||
### Enable High-cardinality Labels for Metrics
|
### Enable High-cardinality Labels for Metrics
|
||||||
|
|
||||||
Enable high-cardinality labels for [Prometheus Metrics](/enterprise/metrics.mdx). Enabling this setting will add more labels to
|
Enable high-cardinality labels for [Prometheus Metrics](/observability/prometheus/metrics.mdx). Enabling this setting
|
||||||
some of the metrics (e.g. `operation_name` label for Graphql subscription metrics).
|
will add more labels to some of the metrics (e.g. `operation_name` label for Graphql subscription metrics).
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------------------- | ------------------------------------------------- |
|
| ------------------- | ------------------------------------------------------- |
|
||||||
| **Flag** | N/A |
|
| **Flag** | N/A |
|
||||||
| **Env var** | `HASURA_GRAPHQL_METRICS_ENABLE_HIGH_CARDINALITY_LABELS` |
|
| **Env var** | `HASURA_GRAPHQL_METRICS_ENABLE_HIGH_CARDINALITY_LABELS` |
|
||||||
| **Accepted values** | Boolean |
|
| **Accepted values** | Boolean |
|
||||||
| **Options** | `true` or `false` |
|
| **Options** | `true` or `false` |
|
||||||
| **Default** | `true` |
|
| **Default** | `true` |
|
||||||
| **Supported in** | Enterprise Edition |
|
| **Supported in** | Enterprise Edition |
|
||||||
|
|
||||||
### Enable Log Compression
|
### Enable Log Compression
|
||||||
|
|
||||||
Enable sending compressed logs to [metrics server](/enterprise/metrics.mdx).
|
Enable sending compressed logs to [metrics server](/observability/prometheus/metrics.mdx).
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------------------- | ------------------------------------------ |
|
| ------------------- | ------------------------------------------ |
|
||||||
@ -614,12 +614,12 @@ This variable sets the level for [Hasura's logs](/deployment/logging.mdx#logging
|
|||||||
|
|
||||||
The [maximum cache size](/enterprise/caching.mdx), measured in MB, for queries.
|
The [maximum cache size](/enterprise/caching.mdx), measured in MB, for queries.
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------------------- | --------------------------------------------------------------------------------- |
|
| ------------------- | ------------------------------------------------------------------------------------------------- |
|
||||||
| **Flag** | `--max-cache-size <SIZE_IN_MB>` |
|
| **Flag** | `--max-cache-size <SIZE_IN_MB>` |
|
||||||
| **Env var** | `HASURA_GRAPHQL_MAX_CACHE_SIZE` |
|
| **Env var** | `HASURA_GRAPHQL_MAX_CACHE_SIZE` |
|
||||||
| **Accepted values** | Integer (Representing cache size measured in MB) |
|
| **Accepted values** | Integer (Representing cache size measured in MB) |
|
||||||
| **Default** | `1` |
|
| **Default** | `1` |
|
||||||
| **Supported in** | **Enterprise Edition**, **Cloud**: Standard / Professional tier is set to `100` MB as the default |
|
| **Supported in** | **Enterprise Edition**, **Cloud**: Standard / Professional tier is set to `100` MB as the default |
|
||||||
|
|
||||||
### Metadata Database Extension Schema
|
### Metadata Database Extension Schema
|
||||||
@ -873,8 +873,8 @@ The path to a shared CA store to use to connect to both (caching and rate-limiti
|
|||||||
|
|
||||||
### Redis URL
|
### Redis URL
|
||||||
|
|
||||||
The Redis URL to use for [query caching](/enterprise/caching.mdx) and [Webhook Auth
|
The Redis URL to use for [query caching](/enterprise/caching.mdx) and
|
||||||
Caching](/auth/authentication/webhook.mdx#webhook-auth-caching).
|
[Webhook Auth Caching](/auth/authentication/webhook.mdx#webhook-auth-caching).
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------------------- | ---------------------------------------- |
|
| ------------------- | ---------------------------------------- |
|
||||||
@ -939,13 +939,13 @@ List of third-party identity providers to enable Single Sign-on authentication f
|
|||||||
Multiplexed [streaming queries](/subscriptions/postgres/streaming/index.mdx) are split into batches of the specified
|
Multiplexed [streaming queries](/subscriptions/postgres/streaming/index.mdx) are split into batches of the specified
|
||||||
size.
|
size.
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------------------- | ------------------------------------------------------------------------- |
|
| ------------------- | --------------------------------------------------------- |
|
||||||
| **Flag** | `--streaming-queries-multiplexed-batch-size <SIZE>` |
|
| **Flag** | `--streaming-queries-multiplexed-batch-size <SIZE>` |
|
||||||
| **Env var** | `HASURA_GRAPHQL_STREAMING_QUERIES_MULTIPLEXED_BATCH_SIZE` |
|
| **Env var** | `HASURA_GRAPHQL_STREAMING_QUERIES_MULTIPLEXED_BATCH_SIZE` |
|
||||||
| **Accepted values** | Integer |
|
| **Accepted values** | Integer |
|
||||||
| **Default** | `100` |
|
| **Default** | `100` |
|
||||||
| **Supported in** | CE, Enterprise Edition |
|
| **Supported in** | CE, Enterprise Edition |
|
||||||
|
|
||||||
### Streaming Queries Multiplexed Refetch Interval
|
### Streaming Queries Multiplexed Refetch Interval
|
||||||
|
|
||||||
@ -1023,7 +1023,7 @@ This identifies an [unauthorized role](/auth/authentication/unauthenticated-acce
|
|||||||
| **Accepted values** | String |
|
| **Accepted values** | String |
|
||||||
| **Example** | Setting this value to `anonymous`, whenever the `Authorization` header is absent, the request's role will default to `anonymous`. |
|
| **Example** | Setting this value to `anonymous`, whenever the `Authorization` header is absent, the request's role will default to `anonymous`. |
|
||||||
| **Default** | `null` |
|
| **Default** | `null` |
|
||||||
| **Supported in** | CE, Enterprise Edition, Cloud |
|
| **Supported in** | CE, Enterprise Edition, Cloud |
|
||||||
|
|
||||||
### Use Prepared Statements
|
### Use Prepared Statements
|
||||||
|
|
||||||
|
@ -33,4 +33,4 @@ When you are ready to move Hasura to production, check out our
|
|||||||
continuous availability.
|
continuous availability.
|
||||||
- We recommend running Hasura with at least 4 CPU cores and a minimum of 8 GB RAM in production. Please set autoscaling
|
- We recommend running Hasura with at least 4 CPU cores and a minimum of 8 GB RAM in production. Please set autoscaling
|
||||||
on CPU.
|
on CPU.
|
||||||
- [Enable and consume metrics](/enterprise/metrics.mdx/#enable-metrics-endpoint).
|
- [Enable and consume metrics](/observability/prometheus/index.mdx).
|
||||||
|
@ -165,7 +165,7 @@ import Enterprise from '@site/static/icons/features/enterprise.svg';
|
|||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
</VersionedLink>
|
</VersionedLink>
|
||||||
<VersionedLink to="/enterprise/metrics/">
|
<VersionedLink to="/observability/prometheus/index/">
|
||||||
<div className="card">
|
<div className="card">
|
||||||
<h3>Metrics via Prometheus</h3>
|
<h3>Metrics via Prometheus</h3>
|
||||||
<p>Learn how to configure Prometheus in Hasura Enterprise Edition to monitor your GraphQL API.</p>
|
<p>Learn how to configure Prometheus in Hasura Enterprise Edition to monitor your GraphQL API.</p>
|
||||||
|
@ -50,8 +50,8 @@ events queue to the webhook.
|
|||||||
|
|
||||||
<ProductBadge self />
|
<ProductBadge self />
|
||||||
|
|
||||||
Hasura exposes a set of [Prometheus metrics](/enterprise/metrics.mdx) that can be used to monitor the Event Trigger
|
Hasura exposes a set of [Prometheus metrics](/observability/prometheus/metrics.mdx) that can be used to monitor the
|
||||||
system and help diagnose performance issues.
|
Event Trigger system and help diagnose performance issues.
|
||||||
|
|
||||||
### Event fetch time per batch
|
### Event fetch time per batch
|
||||||
|
|
||||||
|
@ -1,5 +1,5 @@
|
|||||||
{
|
{
|
||||||
"label": "Integrations",
|
"label": "Integrations",
|
||||||
"position": 2,
|
"position": 3,
|
||||||
"className": "cloud-icon"
|
"className": "cloud-icon"
|
||||||
}
|
}
|
||||||
|
@ -222,5 +222,5 @@ choice:
|
|||||||
- [Datadog](/observability/integrations/datadog.mdx)
|
- [Datadog](/observability/integrations/datadog.mdx)
|
||||||
- [New Relic](/observability/integrations/newrelic.mdx)
|
- [New Relic](/observability/integrations/newrelic.mdx)
|
||||||
- [Azure monitor](/observability/integrations/azure-monitor.mdx)
|
- [Azure monitor](/observability/integrations/azure-monitor.mdx)
|
||||||
- [Prometheus](/observability/integrations/prometheus.mdx)
|
- [Prometheus](/observability/prometheus/cloud-integration.mdx)
|
||||||
- [OpenTelemetry](/observability/integrations/opentelemetry.mdx)
|
- [OpenTelemetry](/observability/integrations/opentelemetry.mdx)
|
||||||
|
@ -35,13 +35,10 @@ import Observability from '@site/static/icons/features/observability.svg';
|
|||||||
.
|
.
|
||||||
</p>
|
</p>
|
||||||
<p>
|
<p>
|
||||||
For our Enterprise customers, we have a set of pre-built dashboards and alerting rules configured with thew Prometheus Grafana Jaeger
|
For our Enterprise customers, we have a set of pre-built dashboards and alerting rules configured with thew
|
||||||
stack, with which you can monitor and debug Hasura. These dashboards will be available soon and integrated with Hasura Cloud
|
Prometheus Grafana Jaeger stack, with which you can monitor and debug Hasura. These dashboards will be available
|
||||||
too. You can read more and explore these dashboards
|
soon and integrated with Hasura Cloud too. You can read more and explore these dashboards
|
||||||
<VersionedLink to="/observability/pre-built-dashboards/">
|
<VersionedLink to="/observability/prometheus/pre-built-dashboards/">here</VersionedLink>.
|
||||||
here
|
|
||||||
</VersionedLink>
|
|
||||||
.
|
|
||||||
</p>
|
</p>
|
||||||
<h4>Quick Links</h4>
|
<h4>Quick Links</h4>
|
||||||
<ul>
|
<ul>
|
||||||
@ -52,7 +49,7 @@ import Observability from '@site/static/icons/features/observability.svg';
|
|||||||
<VersionedLink to="/observability/how-it-works">Learn how Observability works.</VersionedLink>
|
<VersionedLink to="/observability/how-it-works">Learn how Observability works.</VersionedLink>
|
||||||
</li>
|
</li>
|
||||||
<li>
|
<li>
|
||||||
<VersionedLink to="/observability/pre-built-dashboards">Pre-built dashboards.</VersionedLink>
|
<VersionedLink to="/observability/prometheus/pre-built-dashboards">Pre-built dashboards.</VersionedLink>
|
||||||
</li>
|
</li>
|
||||||
</ul>
|
</ul>
|
||||||
</div>
|
</div>
|
||||||
@ -73,7 +70,7 @@ import Observability from '@site/static/icons/features/observability.svg';
|
|||||||
<p>Connect your Hasura GraphQL API to OpenTelemetry-compliant services.</p>
|
<p>Connect your Hasura GraphQL API to OpenTelemetry-compliant services.</p>
|
||||||
</div>
|
</div>
|
||||||
</VersionedLink>
|
</VersionedLink>
|
||||||
<VersionedLink to="/observability/integrations/prometheus/">
|
<VersionedLink to="/observability/prometheus/cloud-integration/">
|
||||||
<div className="card">
|
<div className="card">
|
||||||
<h3>Prometheus</h3>
|
<h3>Prometheus</h3>
|
||||||
<p>Connect your Hasura GraphQL API to Prometheus.</p>
|
<p>Connect your Hasura GraphQL API to Prometheus.</p>
|
||||||
|
5
docs/docs/observability/prometheus/_category_.json
Normal file
5
docs/docs/observability/prometheus/_category_.json
Normal file
@ -0,0 +1,5 @@
|
|||||||
|
{
|
||||||
|
"label": "Prometheus",
|
||||||
|
"position": 2,
|
||||||
|
"className": "cloud-and-enterprise-icon"
|
||||||
|
}
|
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
sidebar_label: Prometheus
|
sidebar_label: Integrate with Hasura Cloud
|
||||||
sidebar_position: 4
|
sidebar_position: 3
|
||||||
description: Prometheus Integration on Hasura Cloud
|
description: Prometheus Integration on Hasura Cloud
|
||||||
title: 'Cloud: Prometheus Integration'
|
title: 'Cloud: Prometheus Integration'
|
||||||
keywords:
|
keywords:
|
||||||
@ -18,7 +18,7 @@ import Thumbnail from '@site/src/components/Thumbnail';
|
|||||||
import HeadingIcon from '@site/src/components/HeadingIcon';
|
import HeadingIcon from '@site/src/components/HeadingIcon';
|
||||||
import ProductBadge from '@site/src/components/ProductBadge';
|
import ProductBadge from '@site/src/components/ProductBadge';
|
||||||
|
|
||||||
# Prometheus Integration
|
# Prometheus Integration for Hasura Cloud
|
||||||
|
|
||||||
<ProductBadge standard pro ee />
|
<ProductBadge standard pro ee />
|
||||||
|
|
353
docs/docs/observability/prometheus/grafana-dashboard.mdx
Normal file
353
docs/docs/observability/prometheus/grafana-dashboard.mdx
Normal file
@ -0,0 +1,353 @@
|
|||||||
|
---
|
||||||
|
sidebar_position: 2
|
||||||
|
sidebar_label: Integrate with Hasura EE
|
||||||
|
title: 'Integrate Prometheus with Hasura EE'
|
||||||
|
description: Install Prometheus server and Grafana to create a basic observability dashboard for Hasura.
|
||||||
|
keywords:
|
||||||
|
- hasura
|
||||||
|
- docs
|
||||||
|
- cloud
|
||||||
|
- integrations
|
||||||
|
- exporter
|
||||||
|
- integration
|
||||||
|
- observability
|
||||||
|
- prometheus
|
||||||
|
- grafana
|
||||||
|
- monitoring
|
||||||
|
- monitoring framework
|
||||||
|
---
|
||||||
|
|
||||||
|
import HeadingIcon from '@site/src/components/HeadingIcon';
|
||||||
|
import ProductBadge from '@site/src/components/ProductBadge';
|
||||||
|
import Thumbnail from '@site/src/components/Thumbnail';
|
||||||
|
|
||||||
|
# Integrate Prometheus with Hasura EE and build a Grafana Dashboard
|
||||||
|
|
||||||
|
<ProductBadge self />
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This guide will help you set up a basic observability dashboard for Hasura using Prometheus and Grafana. We have two
|
||||||
|
approaches depending on your use case:
|
||||||
|
|
||||||
|
- **Self-hosted**: If you are running Prometheus and Grafana on your own infrastructure, follow the
|
||||||
|
[self-hosted installation](#self-hosted-installation) instructions.
|
||||||
|
- **Containerized**: If you are running Prometheus and Grafana in a containerized environment, follow the
|
||||||
|
[containerized installation](#containerized-installation) instructions.
|
||||||
|
|
||||||
|
## Self-hosted installation
|
||||||
|
|
||||||
|
### Install and configure Prometheus
|
||||||
|
|
||||||
|
#### Step 1. Set up the environment
|
||||||
|
|
||||||
|
You will need to create a Prometheus user and group, and a directory for Prometheus to store its data. You will also
|
||||||
|
need to create a directory for Prometheus to store its configuration files.
|
||||||
|
|
||||||
|
This section is written based on an Ubuntu/Debian installation environment. The following commands will help you prepare
|
||||||
|
your environment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo groupadd -system prometheus
|
||||||
|
sudo useradd -s /sbin/nologin -system -g prometheus prometheus
|
||||||
|
sudo mkdir /var/lib/prometheus
|
||||||
|
for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 2. Install Prometheus
|
||||||
|
|
||||||
|
The following set of commands will help you download and install Prometheus:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt update
|
||||||
|
sudo apt -y install wget curl
|
||||||
|
mkdir -p /tmp/prometheus && cd /tmp/prometheus
|
||||||
|
curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest |
|
||||||
|
grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -
|
||||||
|
tar xvf prometheus*.tar.gz
|
||||||
|
cd prometheus*/
|
||||||
|
sudo mv prometheus promtool /usr/local/bin/
|
||||||
|
```
|
||||||
|
|
||||||
|
You can check to see if Prometheus is installed correctly by running the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
prometheus --version
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 3. Connect Prometheus to Hasura
|
||||||
|
|
||||||
|
To connect Prometheus to Hasura, you will need to create a configuration file for Prometheus. The following commands
|
||||||
|
will help you do this:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo cp -rpf prometheus.yml /etc/prometheus/prometheus.yml sudo mv consoles/ console_libraries/ /etc/prometheus/
|
||||||
|
```
|
||||||
|
|
||||||
|
Then, you'll need to edit the Prometheus configuration file (`/etc/prometheus/prometheus.yml`) to include the changes
|
||||||
|
listed below:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# my global config
|
||||||
|
global:
|
||||||
|
scrape_interval: 15s
|
||||||
|
evaluation_interval: 15s
|
||||||
|
|
||||||
|
# scrape_timeout is set to the global default (10s).
|
||||||
|
|
||||||
|
# Alertmanager configuration
|
||||||
|
alerting:
|
||||||
|
alertmanagers:
|
||||||
|
- static_configs:
|
||||||
|
- targets:
|
||||||
|
# - alertmanager:9093
|
||||||
|
|
||||||
|
# Load rules once and periodically evaluate them according to the global ’evaluation_interval ’.
|
||||||
|
rule_files:
|
||||||
|
# - "first_rules.yml"
|
||||||
|
# - "second_rules.yml"
|
||||||
|
|
||||||
|
# A scrape configuration containing exactly one endpoint to scrape:
|
||||||
|
# Here it’s Prometheus itself.
|
||||||
|
scrape_configs:
|
||||||
|
# The job name is added as a label ‘job=<job_name>‘ to any timeseries scraped from this config.
|
||||||
|
- job_name: 'prometheus'
|
||||||
|
# metrics_path defaults to ’/metrics ’
|
||||||
|
# scheme defaults to ’http ’.
|
||||||
|
static_configs:
|
||||||
|
- targets: ['localhost:9090']
|
||||||
|
- job_name: 'graphQL'
|
||||||
|
metrics_path: 'v1/metrics'
|
||||||
|
# metrics_path defaults to ’/metrics ’
|
||||||
|
# scheme defaults to ’http ’.
|
||||||
|
static_configs:
|
||||||
|
- targets: ['hasura_deployment_url:8080']
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 4. Set firewall rules
|
||||||
|
|
||||||
|
If you are using a firewall, you will need to set the following rules:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo ufw allow 9090/tcp
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 5. Set up a password for Prometheus web access
|
||||||
|
|
||||||
|
To set up a password for Prometheus web access, you will need to create a hashed password. First, we'll create the YAML
|
||||||
|
file which will store the password. Inside `/etc/prometheus/`, run the following:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo touch web.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Then, we'll install bcrypt:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt install python3-bcrypt -y
|
||||||
|
```
|
||||||
|
|
||||||
|
Then, we'll create a hashed password via a Python script called `genpass.py` which we can store anywhere:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import getpass
|
||||||
|
import bcrypt
|
||||||
|
|
||||||
|
password = getpass.getpass("password: ")
|
||||||
|
hashed_password = bcrypt.hashpw(password.encode("utf-8"), bcrypt.gensalt())
|
||||||
|
print(hashed_password.decode())
|
||||||
|
```
|
||||||
|
|
||||||
|
You can then run the script, using the command below, and enter your password when prompted:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 gen-pass.py
|
||||||
|
```
|
||||||
|
|
||||||
|
The output will be a hashed password. Copy this password and paste it into the `web.yml` file, as shown below:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
basic_auth_users:
|
||||||
|
admin: ’your new hashed value ’
|
||||||
|
```
|
||||||
|
|
||||||
|
To check yourself, use `promtool` to check the configuration file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
promtool check web-config /etc/prometheus/web.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 6. Restart Prometheus
|
||||||
|
|
||||||
|
To restart Prometheus, run the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo systemctl restart prometheus
|
||||||
|
```
|
||||||
|
|
||||||
|
Then, test the password by running:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -u admin:<YOUR_PASSWORD> http://localhost:9090/metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see a response similar to the one below:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
|
||||||
|
# TYPE go_gc_duration_seconds summary
|
||||||
|
go_gc_duration_seconds{quantile="0"} 0
|
||||||
|
go_gc_duration_seconds{quantile="0.25"} 0
|
||||||
|
# etc...
|
||||||
|
```
|
||||||
|
|
||||||
|
### Install and configure Grafana
|
||||||
|
|
||||||
|
#### Step 7. Install Grafana
|
||||||
|
|
||||||
|
You can install Grafana by running the following commands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
|
||||||
|
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main" sudo apt update
|
||||||
|
sudo apt install grafana
|
||||||
|
sudo systemctl start grafana-server
|
||||||
|
sudo systemctl enable grafana-server
|
||||||
|
```
|
||||||
|
|
||||||
|
At this point, your Grafana server should be available at `http://<YOUR_IP_ADDRESS>:3000` where you'll find the login
|
||||||
|
screen. The default username and password are both `admin`.
|
||||||
|
|
||||||
|
:::info Change the default password
|
||||||
|
|
||||||
|
After logging in, you will be prompted to change the default password. Set your new password and login.
|
||||||
|
|
||||||
|
:::
|
||||||
|
|
||||||
|
#### Step 8. Create a Prometheus data source
|
||||||
|
|
||||||
|
In Grafana, from the settings icon on the sidebar, open the `Configuration` menu and select `Data Sources`. Then, click
|
||||||
|
on `Add data source` and select `Prometheus` as the type.
|
||||||
|
|
||||||
|
Then, set the appropriate URL for your Prometheus server (e.g., `http://localhost:9090`) and click `Save & Test`. If
|
||||||
|
everything is working correctly, you should see a green `Data source is working` message.
|
||||||
|
|
||||||
|
<Thumbnail
|
||||||
|
src="/img/enterprise/prometheus/create-prometheus-data-source.png"
|
||||||
|
alt="Create Prometheus data source"
|
||||||
|
width="1000px"
|
||||||
|
/>
|
||||||
|
|
||||||
|
#### Step 9. Create a Prometheus graph
|
||||||
|
|
||||||
|
Click the graph title and select `Edit`. Then, select the `Metrics` tab and select your Prometheus data source. Then,
|
||||||
|
enter any Prometheus expression ino the `Query` field while using the `Metric` field to lookup metrics via autocomplete.
|
||||||
|
|
||||||
|
<Thumbnail
|
||||||
|
src="/img/enterprise/prometheus/create-a-prometheus-graph.png"
|
||||||
|
alt="Create Prometheus data source"
|
||||||
|
width="1000px"
|
||||||
|
/>
|
||||||
|
|
||||||
|
:::info Formatting legend names
|
||||||
|
|
||||||
|
To format the legend names of time series, use the "Legend format" input. For example, to show only the method and
|
||||||
|
status labels of a returned query result, separated by a dash, you could use the legend format string
|
||||||
|
`{{method}} - {{status}}`.
|
||||||
|
|
||||||
|
<Thumbnail
|
||||||
|
src="/img/enterprise/prometheus/formatting-legend-names.png"
|
||||||
|
alt="Create Prometheus data source"
|
||||||
|
width="1000px"
|
||||||
|
/>
|
||||||
|
|
||||||
|
:::
|
||||||
|
|
||||||
|
## Containerized installation
|
||||||
|
|
||||||
|
### Install and configure Prometheus and Grafana
|
||||||
|
|
||||||
|
#### Step 1. Prepare the Prometheus configuration file
|
||||||
|
|
||||||
|
Create a file named `prometheus.yml` on your host with the following information:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# my global config
|
||||||
|
global:
|
||||||
|
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
|
||||||
|
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
|
||||||
|
# scrape_timeout is set to the global default (10s).
|
||||||
|
|
||||||
|
# Alertmanager configuration
|
||||||
|
alerting:
|
||||||
|
alertmanagers:
|
||||||
|
- static_configs:
|
||||||
|
- targets:
|
||||||
|
# - alertmanager:9093
|
||||||
|
|
||||||
|
# Load rules once and periodically evaluate them according to the global ’evaluation_interval ’.
|
||||||
|
rule_files:
|
||||||
|
# - "first_rules.yml"
|
||||||
|
# - "second_rules.yml"
|
||||||
|
|
||||||
|
# A scrape configuration containing exactly one endpoint to scrape:
|
||||||
|
# Here it’s Prometheus itself.
|
||||||
|
scrape_configs:
|
||||||
|
- job_name: 'prometheus'
|
||||||
|
static_configs:
|
||||||
|
- targets: ['localhost:9090']
|
||||||
|
- job_name: 'hasura'
|
||||||
|
metrics_path: '/v1/metrics'
|
||||||
|
static_configs:
|
||||||
|
- targets: ['ip_address_of_hasura_installation:8080']
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 2. Pull the Prometheus and Grafana Docker containers
|
||||||
|
|
||||||
|
For Prometheus, run the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run -p 9090:9090 -v /path/to/your/local/prometheus.yml:/etc/ prometheus/prometheus.yml prom/prometheus
|
||||||
|
```
|
||||||
|
|
||||||
|
Then, for Grafana, run the following:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run -d -p 3000:3000 grafana/grafana-enterprise
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configure Grafana
|
||||||
|
|
||||||
|
#### Step 3. Adding a Prometheus as a data source in Grafana
|
||||||
|
|
||||||
|
In Grafana, from the settings icon on the sidebar, open the `Configuration` menu and select `Data Sources`. Then, click
|
||||||
|
on `Add data source` and select `Prometheus` as the type.
|
||||||
|
|
||||||
|
Then, set the appropriate URL for your Prometheus server (e.g., `http://localhost:9090`) and click `Save & Test`. If
|
||||||
|
everything is working correctly, you should see a green `Alerting supported` message.
|
||||||
|
|
||||||
|
<Thumbnail
|
||||||
|
src="/img/enterprise/prometheus/create-prometheus-data-source.png"
|
||||||
|
alt="Create Prometheus data source"
|
||||||
|
width="1000px"
|
||||||
|
/>
|
||||||
|
|
||||||
|
#### Step 5. Add Hasura metrics to the dashboard
|
||||||
|
|
||||||
|
Click on the `Add Panel` icon in the top-right corner of the Grafana dashboard. Then, select `Add New Panel` or
|
||||||
|
`Add New Row`.
|
||||||
|
|
||||||
|
<Thumbnail
|
||||||
|
src="/img/enterprise/prometheus/add-metrics-to-dashboard.png"
|
||||||
|
alt="Create Prometheus data source"
|
||||||
|
width="1000px"
|
||||||
|
/>
|
||||||
|
|
||||||
|
Click on the `Metric` section and start typing, `hasura` — you should see a list of available Hasura metrics. Select the
|
||||||
|
metric you want to add to the dashboard.
|
||||||
|
|
||||||
|
<Thumbnail
|
||||||
|
src="/img/enterprise/prometheus/create-a-prometheus-graph.png"
|
||||||
|
alt="Create Prometheus data source"
|
||||||
|
width="1000px"
|
||||||
|
/>
|
38
docs/docs/observability/prometheus/index.mdx
Normal file
38
docs/docs/observability/prometheus/index.mdx
Normal file
@ -0,0 +1,38 @@
|
|||||||
|
---
|
||||||
|
slug: index
|
||||||
|
title: 'EE: Integrations with Prometheus'
|
||||||
|
description: Configure integrations with Hasura Enterprise Edition
|
||||||
|
keywords:
|
||||||
|
- hasura
|
||||||
|
- docs
|
||||||
|
- enterprise
|
||||||
|
- ee
|
||||||
|
- integrations
|
||||||
|
- exporter
|
||||||
|
- integration
|
||||||
|
- observability
|
||||||
|
- monitoring
|
||||||
|
- monitoring framework
|
||||||
|
- prometheus
|
||||||
|
---
|
||||||
|
|
||||||
|
import HeadingIcon from '@site/src/components/HeadingIcon';
|
||||||
|
import ProductBadge from '@site/src/components/ProductBadge';
|
||||||
|
|
||||||
|
# Integrate Prometheus with Hasura Enterprise Edition
|
||||||
|
|
||||||
|
<ProductBadge self pro ee />
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
In this section, you'll find information on how to integrate [Prometheus](https://prometheus.io/) with Hasura Enterprise
|
||||||
|
Edition:
|
||||||
|
|
||||||
|
- [Available metrics](/observability/prometheus/metrics.mdx): Learn about metrics available to monitor the health,
|
||||||
|
performance and reliability of the Hasura GraphQL Engine.
|
||||||
|
- [Integrate with Hasura Cloud](/observability/prometheus/cloud-integration.mdx): Configure Prometheus integration with
|
||||||
|
Hasura Enterprise Edition.
|
||||||
|
- [Integrate Prometheus with Hasura EE and build a Grafana Dashboard](/observability/prometheus/grafana-dashboard.mdx):
|
||||||
|
Configure Prometheus integration with Hasura Enterprise Edition.
|
||||||
|
- [Pre-built dashboards](/observability/prometheus/pre-built-dashboards.mdx): Learn about pre-built dashboards available
|
||||||
|
for Hasura Enterprise Edition.
|
@ -1,12 +1,12 @@
|
|||||||
---
|
---
|
||||||
sidebar_label: Metrics via Prometheus
|
sidebar_label: Available Metrics
|
||||||
description: Metrics via Prometheus for Hasura Enterprise Edition
|
description: Metrics via Prometheus for Hasura Enterprise Edition
|
||||||
title: 'Enterprise Edition: Metrics via Prometheus'
|
title: 'Enterprise Edition: Metrics via Prometheus'
|
||||||
keywords:
|
keywords:
|
||||||
- hasura
|
- hasura
|
||||||
- docs
|
- docs
|
||||||
- enterprise
|
- enterprise
|
||||||
sidebar_position: 4
|
sidebar_position: 1
|
||||||
---
|
---
|
||||||
|
|
||||||
import ProductBadge from '@site/src/components/ProductBadge';
|
import ProductBadge from '@site/src/components/ProductBadge';
|
||||||
@ -59,14 +59,14 @@ The following metrics are exported by Hasura GraphQL Engine:
|
|||||||
|
|
||||||
The following metrics can be used to monitor the performance of Hasura Event Triggers system:
|
The following metrics can be used to monitor the performance of Hasura Event Triggers system:
|
||||||
|
|
||||||
- [`hasura_event_fetch_time_per_batch_seconds`](/event-triggers/observability-and-performace.mdx/#event-fetch-time-per-batch)
|
- [`hasura_event_fetch_time_per_batch_seconds`](/event-triggers/observability-and-performace.mdx/#event-fetch-time-per-batch)
|
||||||
- [`hasura_event_invocations_total`](/event-triggers/observability-and-performace.mdx/#event-invocations-total)
|
- [`hasura_event_invocations_total`](/event-triggers/observability-and-performace.mdx/#event-invocations-total)
|
||||||
- [`hasura_event_processed_total`](/event-triggers/observability-and-performace.mdx/#event-processed-total)
|
- [`hasura_event_processed_total`](/event-triggers/observability-and-performace.mdx/#event-processed-total)
|
||||||
- [`hasura_event_processing_time_seconds`](/event-triggers/observability-and-performace.mdx/#event-processing-time)
|
- [`hasura_event_processing_time_seconds`](/event-triggers/observability-and-performace.mdx/#event-processing-time)
|
||||||
- [`hasura_event_queue_time_seconds`](/event-triggers/observability-and-performace.mdx/#event-queue-time)
|
- [`hasura_event_queue_time_seconds`](/event-triggers/observability-and-performace.mdx/#event-queue-time)
|
||||||
- [`hasura_event_trigger_http_workers`](/event-triggers/observability-and-performace.mdx/#event-triggers-http-workers)
|
- [`hasura_event_trigger_http_workers`](/event-triggers/observability-and-performace.mdx/#event-triggers-http-workers)
|
||||||
- [`hasura_event_webhook_processing_time_seconds`](/event-triggers/observability-and-performace.mdx/#event-webhook-processing-time)
|
- [`hasura_event_webhook_processing_time_seconds`](/event-triggers/observability-and-performace.mdx/#event-webhook-processing-time)
|
||||||
- [`hasura_events_fetched_per_batch`](/event-triggers/observability-and-performace.mdx/#events-fetched-per-batch)
|
- [`hasura_events_fetched_per_batch`](/event-triggers/observability-and-performace.mdx/#events-fetched-per-batch)
|
||||||
|
|
||||||
### Subscription Metrics
|
### Subscription Metrics
|
||||||
|
|
||||||
@ -80,7 +80,8 @@ The following metrics can be used to monitor the performance of subscriptions:
|
|||||||
|
|
||||||
### Hasura cache request count
|
### Hasura cache request count
|
||||||
|
|
||||||
Tracks cache hit and miss requests, which helps in monitoring and optimizing cache utilization. You can read more about this [here](/caching/caching-metrics.mdx).
|
Tracks cache hit and miss requests, which helps in monitoring and optimizing cache utilization. You can read more about
|
||||||
|
this [here](/caching/caching-metrics.mdx).
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------ | ---------------------------- |
|
| ------ | ---------------------------- |
|
||||||
@ -115,11 +116,11 @@ webhook.
|
|||||||
Execution time of successful GraphQL requests (excluding subscriptions). If more requests are falling in the higher
|
Execution time of successful GraphQL requests (excluding subscriptions). If more requests are falling in the higher
|
||||||
buckets, you should consider [tuning the performance](/deployment/performance-tuning.mdx).
|
buckets, you should consider [tuning the performance](/deployment/performance-tuning.mdx).
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------ | -------------------------------------------------------------- |
|
| ------ | ------------------------------------------------------------ |
|
||||||
| Name | `hasura_graphql_execution_time_seconds` |
|
| Name | `hasura_graphql_execution_time_seconds` |
|
||||||
| Type | Histogram<br /><br />Buckets: 0.01, 0.03, 0.1, 0.3, 1, 3, 10 |
|
| Type | Histogram<br /><br />Buckets: 0.01, 0.03, 0.1, 0.3, 1, 3, 10 |
|
||||||
| Labels | `operation_type`: query \| mutation |
|
| Labels | `operation_type`: query \| mutation |
|
||||||
|
|
||||||
### Hasura GraphQL requests total
|
### Hasura GraphQL requests total
|
||||||
|
|
||||||
@ -179,16 +180,15 @@ Current number of active PostgreSQL connections. Compare this to
|
|||||||
|
|
||||||
### Hasura source health
|
### Hasura source health
|
||||||
|
|
||||||
Health check status of a particular data source, corresponding to the output of
|
Health check status of a particular data source, corresponding to the output of `/healthz/sources`, with possible values
|
||||||
`/healthz/sources`, with possible values 0 through 3 indicating, respectively:
|
0 through 3 indicating, respectively: OK, TIMEOUT, FAILED, ERROR. See the
|
||||||
OK, TIMEOUT, FAILED, ERROR. See the [Source Health Check API Reference](/api-reference/source-health.mdx)
|
[Source Health Check API Reference](/api-reference/source-health.mdx) for details.
|
||||||
for details.
|
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| ------ | -------------------------------------- |
|
| ------ | ----------------------------------- |
|
||||||
| Name | `hasura_source_health` |
|
| Name | `hasura_source_health` |
|
||||||
| Type | Gauge |
|
| Type | Gauge |
|
||||||
| Labels | `source_name`: name of the database |
|
| Labels | `source_name`: name of the database |
|
||||||
|
|
||||||
### Hasura WebSocket connections
|
### Hasura WebSocket connections
|
||||||
|
|
@ -19,14 +19,16 @@ import Thumbnail from '@site/src/components/Thumbnail';
|
|||||||
import ProductBadge from '@site/src/components/ProductBadge';
|
import ProductBadge from '@site/src/components/ProductBadge';
|
||||||
|
|
||||||
## Subscription Execution
|
## Subscription Execution
|
||||||
|
|
||||||
For serving subscription requests, Hasura optimizes the subscription execution to ensure it is as fast as possible while
|
For serving subscription requests, Hasura optimizes the subscription execution to ensure it is as fast as possible while
|
||||||
not overloading the database with concurrent queries.
|
not overloading the database with concurrent queries.
|
||||||
|
|
||||||
To achieve this, Hasura uses a combination of the following techniques:
|
To achieve this, Hasura uses a combination of the following techniques:
|
||||||
|
|
||||||
- **Grouping same queries into "cohorts" **: Hasura groups subscriptions with the same set of query and session
|
- **Grouping same queries into "cohorts" **: Hasura groups subscriptions with the same set of query and session
|
||||||
variables into a single cohort. The subscribers in a cohort are updated simultaneously.
|
variables into a single cohort. The subscribers in a cohort are updated simultaneously.
|
||||||
- **Diff-checking**: On receiving response from the database, Hasura checks the diff between the old and new values
|
- **Diff-checking**: On receiving response from the database, Hasura checks the diff between the old and new values and
|
||||||
and sends the response only to the subscribers whose values have changed.
|
sends the response only to the subscribers whose values have changed.
|
||||||
- **Multiplexing**: Hasura groups similar "parameterized" subscriptions together and additionally splits them into
|
- **Multiplexing**: Hasura groups similar "parameterized" subscriptions together and additionally splits them into
|
||||||
batches for efficient performance on the database. The batch size can be configured using the
|
batches for efficient performance on the database. The batch size can be configured using the
|
||||||
[`HASURA_GRAPHQL_LIVE_QUERIES_MULTIPLEXED_BATCH_SIZE`](deployment/graphql-engine-flags/reference.mdx#multiplexed-batch-size)
|
[`HASURA_GRAPHQL_LIVE_QUERIES_MULTIPLEXED_BATCH_SIZE`](deployment/graphql-engine-flags/reference.mdx#multiplexed-batch-size)
|
||||||
@ -35,15 +37,15 @@ To achieve this, Hasura uses a combination of the following techniques:
|
|||||||
are grouping the three subscriptions into a single multiplexed query.
|
are grouping the three subscriptions into a single multiplexed query.
|
||||||
|
|
||||||
<Thumbnail
|
<Thumbnail
|
||||||
src='/img/databases/postgres/subscriptions/subscription-multiplexing.png'
|
src="/img/databases/postgres/subscriptions/subscription-multiplexing.png"
|
||||||
alt='Hasura subscription multiplexing AST'
|
alt="Hasura subscription multiplexing AST"
|
||||||
width='900px'
|
width="900px"
|
||||||
className='no-shadow'
|
className="no-shadow"
|
||||||
/>
|
/>
|
||||||
|
|
||||||
For more details on how Hasura executes subscriptions, refer to the [live
|
For more details on how Hasura executes subscriptions, refer to the
|
||||||
query](/subscriptions/postgres/livequery/execution.mdx) or [streaming
|
[live query](/subscriptions/postgres/livequery/execution.mdx) or
|
||||||
subscription](/subscriptions/postgres/streaming/index.mdx) documentation.
|
[streaming subscription](/subscriptions/postgres/streaming/index.mdx) documentation.
|
||||||
|
|
||||||
## Observability
|
## Observability
|
||||||
|
|
||||||
@ -137,7 +139,6 @@ some extra process time for other tasks the poller does during a single poll. In
|
|||||||
| Type | Histogram<br /><br />Buckets: 0.000001, 0.0001, 0.01, 0.1, 0.3, 1, 3, 10, 30, 100 |
|
| Type | Histogram<br /><br />Buckets: 0.000001, 0.0001, 0.01, 0.1, 0.3, 1, 3, 10, 30, 100 |
|
||||||
| Labels | `subscription_kind`: streaming \| live-query, `operation_name`, `parameterized_query_hash` |
|
| Labels | `subscription_kind`: streaming \| live-query, `operation_name`, `parameterized_query_hash` |
|
||||||
|
|
||||||
|
|
||||||
### Subscription Database Execution Time
|
### Subscription Database Execution Time
|
||||||
|
|
||||||
The time taken to run the subscription's multiplexed query in the database for a single batch.
|
The time taken to run the subscription's multiplexed query in the database for a single batch.
|
||||||
@ -159,9 +160,9 @@ consider investigating the subscription query and see if indexes can help improv
|
|||||||
|
|
||||||
## Golden Signals for subscriptions
|
## Golden Signals for subscriptions
|
||||||
|
|
||||||
You can perform [Golden
|
You can perform
|
||||||
Signals-based](https://sre.google/sre-book/monitoring-distributed-systems/#xref_monitoring_golden-signals) system
|
[Golden Signals-based](https://sre.google/sre-book/monitoring-distributed-systems/#xref_monitoring_golden-signals)
|
||||||
monitoring with Hasura's exported metrics. The following are the golden signals for subscriptions:
|
system monitoring with Hasura's exported metrics. The following are the golden signals for subscriptions:
|
||||||
|
|
||||||
### Latency
|
### Latency
|
||||||
|
|
||||||
@ -170,8 +171,9 @@ latency, you can monitor the [`hasura_subscription_total_time_seconds`](#subscri
|
|||||||
|
|
||||||
If the value of this metric is high, then it may be an indication that the multiplexed query is taking longer to execute
|
If the value of this metric is high, then it may be an indication that the multiplexed query is taking longer to execute
|
||||||
in the database, verify this with
|
in the database, verify this with
|
||||||
[`hasura_subscription_db_execution_time_seconds`](#subscription-database-execution-time)
|
[`hasura_subscription_db_execution_time_seconds`](#subscription-database-execution-time) metric. If the value of this
|
||||||
metric. If the value of this metric is high as well, you can do the following:
|
metric is high as well, you can do the following:
|
||||||
|
|
||||||
- Check if any database index can help improve the performance of the query, [analyzing the GraphQL query](#analyze)
|
- Check if any database index can help improve the performance of the query, [analyzing the GraphQL query](#analyze)
|
||||||
will show the generated multiplexed query.
|
will show the generated multiplexed query.
|
||||||
- Avoid querying unnecessary fields that translate to joins or function calls in the GraphQL query.
|
- Avoid querying unnecessary fields that translate to joins or function calls in the GraphQL query.
|
||||||
@ -188,7 +190,8 @@ number of Hasura instances to handle the load.
|
|||||||
### Errors
|
### Errors
|
||||||
|
|
||||||
Errors in subscriptions can be monitored using the following metrics
|
Errors in subscriptions can be monitored using the following metrics
|
||||||
- [`hasura_graphql_requests_total{type="subscription",response_status="error"}`](/enterprise/metrics.mdx#hasura-graphql-requests-total):
|
|
||||||
|
- [`hasura_graphql_requests_total{type="subscription",response_status="error"}`](/observability/prometheus/metrics.mdx#hasura-graphql-requests-total):
|
||||||
Total number of errors that happen before the subscription is started (i.e. validation, parsing and authorization
|
Total number of errors that happen before the subscription is started (i.e. validation, parsing and authorization
|
||||||
errors).
|
errors).
|
||||||
- [`hasura_active_subscription_pollers_in_error_state`](#active-subscription-pollers-in-error-state): Number of
|
- [`hasura_active_subscription_pollers_in_error_state`](#active-subscription-pollers-in-error-state): Number of
|
||||||
@ -204,10 +207,11 @@ Saturation is the threshold until which the subscriptions can run smoothly; once
|
|||||||
performance issues, and abrupt disconnections of the connected subscribers.
|
performance issues, and abrupt disconnections of the connected subscribers.
|
||||||
|
|
||||||
To monitor the saturation for subscriptions, you can monitor the following:
|
To monitor the saturation for subscriptions, you can monitor the following:
|
||||||
|
|
||||||
- CPU and memory usage of Hasura instances.
|
- CPU and memory usage of Hasura instances.
|
||||||
- For postgres backends, you can monitor the
|
- For postgres backends, you can monitor the
|
||||||
[`hasura_postgres_connections`](/enterprise/metrics.mdx#hasura-postgres-connections) metric to see the number of
|
[`hasura_postgres_connections`](/observability/prometheus/metrics.mdx#hasura-postgres-connections) metric to see the
|
||||||
connections opened by Hasura with the database.
|
number of connections opened by Hasura with the database.
|
||||||
- P99 of the [`hasura_subscription_total_time_seconds`](#subscription-total-time) metric.
|
- P99 of the [`hasura_subscription_total_time_seconds`](#subscription-total-time) metric.
|
||||||
|
|
||||||
If the number of database connections is high, you can consider increasing the `max_connections` of the database.
|
If the number of database connections is high, you can consider increasing the `max_connections` of the database.
|
||||||
@ -222,9 +226,9 @@ to analyze and tune the performance of subscriptions.
|
|||||||
|
|
||||||
### Analyze
|
### Analyze
|
||||||
|
|
||||||
Using the `Analyze` button on GraphiQL API Explorer of the Hasura Console, you can see the generated
|
Using the `Analyze` button on GraphiQL API Explorer of the Hasura Console, you can see the generated multiplexed SQL and
|
||||||
multiplexed SQL and its query plan. The query plan can reveal the bottlenecks in query execution and appropriate indexes
|
its query plan. The query plan can reveal the bottlenecks in query execution and appropriate indexes can be added to
|
||||||
can be added to improve performance.
|
improve performance.
|
||||||
|
|
||||||
In addition to these, simplifying the subscription to avoid unnecessary joins or avoiding fetching fields which are not
|
In addition to these, simplifying the subscription to avoid unnecessary joins or avoiding fetching fields which are not
|
||||||
going to change can also help improve performance.
|
going to change can also help improve performance.
|
||||||
@ -236,20 +240,20 @@ The parameters governing the performance of subscriptions in terms of throughput
|
|||||||
- `HASURA_GRAPHQL_LIVE_QUERIES_MULTIPLEXED_REFETCH_INTERVAL`
|
- `HASURA_GRAPHQL_LIVE_QUERIES_MULTIPLEXED_REFETCH_INTERVAL`
|
||||||
- Time interval between Hasura multiplexed queries to the DB for fetching changes in subscriptions data, default = 1
|
- Time interval between Hasura multiplexed queries to the DB for fetching changes in subscriptions data, default = 1
|
||||||
sec
|
sec
|
||||||
- Increasing this reduces the frequency of queries to the DB, thereby reducing its load and improving throughput
|
- Increasing this reduces the frequency of queries to the DB, thereby reducing its load and improving throughput while
|
||||||
while increasing the latency of updates to the clients.
|
increasing the latency of updates to the clients.
|
||||||
- `HASURA_GRAPHQL_LIVE_QUERIES_MULTIPLEXED_BATCH_SIZE`
|
- `HASURA_GRAPHQL_LIVE_QUERIES_MULTIPLEXED_BATCH_SIZE`
|
||||||
- Number of similar subscriptions batched into a single SQL query to the DB, default = 100
|
- Number of similar subscriptions batched into a single SQL query to the DB, default = 100
|
||||||
- Increasing this reduces the number of SQL statements fired to the DB, thereby reducing its load and improving
|
- Increasing this reduces the number of SQL statements fired to the DB, thereby reducing its load and improving
|
||||||
throughput while increasing individual SQL execution times and latency.
|
throughput while increasing individual SQL execution times and latency.
|
||||||
- You should reduce this value if the execution time of the SQL generated by Hasura after multiplexing is more
|
- You should reduce this value if the execution time of the SQL generated by Hasura after multiplexing is more than
|
||||||
than the refetch interval.
|
the refetch interval.
|
||||||
- `max_connections` of the source
|
- `max_connections` of the source
|
||||||
- Max number of connections Hasura opens with the DB, default = 50 (configurable via [update source metadata
|
- Max number of connections Hasura opens with the DB, default = 50 (configurable via
|
||||||
API](api-reference/metadata-api/source.mdx))
|
[update source metadata API](api-reference/metadata-api/source.mdx))
|
||||||
- Increasing this increases the number of connections Hasura can open with the DB to execute queries, thereby
|
- Increasing this increases the number of connections Hasura can open with the DB to execute queries, thereby
|
||||||
improving concurrency **but adding load to the database**. A very high number of concurrent connections can result in poor
|
improving concurrency **but adding load to the database**. A very high number of concurrent connections can result
|
||||||
DB performance.
|
in poor DB performance.
|
||||||
|
|
||||||
:::info Note
|
:::info Note
|
||||||
|
|
||||||
|
BIN
docs/static/img/enterprise/prometheus/add-metrics-to-dashboard.png
vendored
Normal file
BIN
docs/static/img/enterprise/prometheus/add-metrics-to-dashboard.png
vendored
Normal file
Binary file not shown.
After (image error) Size: 146 KiB |
BIN
docs/static/img/enterprise/prometheus/create-a-prometheus-graph.png
vendored
Normal file
BIN
docs/static/img/enterprise/prometheus/create-a-prometheus-graph.png
vendored
Normal file
Binary file not shown.
After (image error) Size: 98 KiB |
BIN
docs/static/img/enterprise/prometheus/create-prometheus-data-source.png
vendored
Normal file
BIN
docs/static/img/enterprise/prometheus/create-prometheus-data-source.png
vendored
Normal file
Binary file not shown.
After (image error) Size: 86 KiB |
BIN
docs/static/img/enterprise/prometheus/formatting-legend-names.png
vendored
Normal file
BIN
docs/static/img/enterprise/prometheus/formatting-legend-names.png
vendored
Normal file
Binary file not shown.
After (image error) Size: 70 KiB |
Loading…
Reference in New Issue
Block a user