ref
https://linear.app/ghost/issue/ENG-1746/enable-ghost-to-push-metrics-to-a-pushgateway
- We'd like to use prometheus to expose metrics from Ghost, but the
"standard" approach of having prometheus scrape the `/metrics` endpoint
adds some complexity and additional challenges on Pro.
- A suggested simpler alternative is to use a pushgateway, to have Ghost
_push_ metrics to prometheus, rather than have prometheus scrape the
running instances.
- This PR introduces this functionality behind a configuration.
- It also includes a refactor to the current metrics-server
implementation so all the related code for prometheus is colocated, and
the configuration is a bit more organized. `@tryghost/metrics-server`
has been renamed to `@tryghost/prometheus-metrics`, and it now includes
the metrics server and prometheus-client code itself (including the
pushgateway code)
- To enable the prometheus client alone, `prometheus:enabled` must be
true. This will _not_ enable the metrics server or the pushgateway — it
will essentially collect the metrics, but not do anything with them.
- To enable the metrics server, set `prometheus:metrics_server:enabled`
to true. You can also configure the host and port that the metrics
server should export the `/metrics` endpoint on in the
`prometheus:metrics_server` block.
- To enable the pushgateway, set `prometheus:pushgateway:enabled` to
true. You can also configure the pushgateway's `url`, the `interval` it
should push metrics in (in milliseconds) and the `jobName` in the
`prometheus:pushgateway` block.
ref
https://linear.app/tryghost/issue/ENG-1591/add-prometheus-and-grafana-services-to-docker-compose
This commit adds 2 new services to the docker compose file to enable
monitoring metrics from Ghost locally in real-time:
1. Prometheus - a service that scrapes Ghost's new `/metrics` endpoint
introduced in this
[commit](768336efad).
2. Grafana - a service that consumes the metrics from prometheus and
exposes them in a dashboard that you can view locally at
`localhost:3000`.
# Usage
Both of these services are selectively enabled using docker compose
[profiles](https://docs.docker.com/compose/how-tos/profiles/). This way,
if you don't opt-in to using these monitoring tools, they won't start
and consume resources on your host machine. To enable these services,
enable the `monitoring` profile by either setting the `COMPOSE_PROFILES`
environment variable to `monitoring`, or specifying the `--profile
monitoring` CLI argument to any `docker compose ...` commands.
I've found the easiest way to configure this in an 'always on' fashion
is to create a `.env` file in the project's root directory and add
`COMPOSE_PROFILES=monitoring` to it. As an added convenience, you can
also set `COMPOSE_FILE=.github/scripts/docker-compose.yml`, which will
allow you to run `docker compose ...` commands from the root directory
without specifying the full path each time.
# Intended for development only
These services are meant for local development only, and are not
configured for a production use-case. For example, the Grafana instance
is configured to have _no authorization_ so you won't need a
username/password to login at `localhost:3000`. Prometheus is also
configured to scrape the metrics once every second, which is likely
excessive for production use-cases, but may be useful for getting more
granular metrics while e.g. load testing locally.
# Dashboards
The Grafana instance includes a default dashboard including most of the
main default metrics provided by our prometheus client integration. The
dashboard is defined in a JSON file at
`.github/scripts/docker/grafana/dashboards/main-dashboard.json' and can
be modified & committed to add new visualizations that will be available
to anyone work on Ghost locally. You can also add other dashboards to
the same directory for specific use-cases, which should be picked up and
made available in the Grafana UI. [Read
more](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/view-dashboard-json-model/)
about Grafana's JSON schema for dashboards.
no issue
- OpenTelemetry has been problematic in a number of ways (boot time,
breaking the frontend). May revisit it at some point in the future, but
for now it is only exporting metrics via prometheus and not traces, so
there's currently nothing sending data to this jaeger container
- Cleaning it up for now as it's just sitting there idly consuming
resources
no issue
- This allows us to run `docker-compose down` or to restart docker
desktop without losing all our local databases
- Added a data volume to the MySQL service in the `docker-compose.yml`
file to persist the data between container restarts
- The `yarn docker:reset` command will still reset all the data in the
database since it uses `down -v` to remove the volumes as well
This commit adds OpenTelemetry instrumentation to Ghost's backend, which
allows us to view traces similar to what we see in Sentry Performance
locally.
OpenTelemetry is enabled if `NODE_ENV === 'development'` or if it is
explicitly enabled via config with `opentelemetry:enabled`.
It also adds a [Jaeger](https://www.jaegertracing.io/) container to
Ghost's docker-compose file for viewing the traces. There's no setup
required (beyond running `yarn docker:reset` to pickup the changes in
the docker-compose file the first time — but this will also reset your
DB so be careful). This will launch the Jaeger container, and you can
view the UI to see the traces at `http://localhost:16686/search`.
no refs
Redis can be utilised for various caching purposes within Ghost. This PR
adds a Redis service to the docker-compose file to allow for easier
local development when Redis is required
no refs
Added a name for the project in `docker-compose.yml` as it was using the
`scripts` directory as the name
Before:
```sh
❯ docker-compose ls
NAME STATUS CONFIG FILES
scripts running(1) /path/to/ghost/.github/scripts/docker-compose.yml
```
After:
```sh
❯ docker-compose ls
NAME STATUS CONFIG FILES
ghost running(1) /path/to/ghost/.github/scripts/docker-compose.yml
```
ref PROD-233
- Stored whether Docker is used in the config files
- When running `yarn setup`, any existing Docker container will be
reset. Run with `-y` to skip the confirmation step.
- `yarn setup` will always init the database and generate fake data
- Increased amount of default generated data to 100,000 members + 500
posts.
- Made lots of performance improvements in the data generator so we can
generate the default data in ±170s
refs https://github.com/TryGhost/DevOps/issues/118
- we should standardize how we run MySQL, so there aren't massive gaps
in developer experience
- Docker is great for this, and it's pretty easy to set up
- this adds a docker-compose.yml file with a small MySQL setup
- also configures `yarn setup` to spin up the Docker container and add
it to config.local.json
- in the near future, we'll provide a base SQL file in .github/scripts/mysql-preload
to ensure the DB is in a known state
