ref
https://linear.app/ghost/issue/ENG-1508/add-custom-metrics-for-email-analytics-jobs
- With the experimental job queue, we're using email analytics as our
initial validation test case. We're hoping to see an improvement in
Ghost's throughput for ingesting email events. However, we don't
currently collect this data point, so it's kind of impossible to tell
right now if the job queue is making things better or not.
- This PR fixes that by adding two new prometheus metrics:
- `email_analytics_events_processed` — a counter incremented each time
an event is processed. Sometimes the event has already been processed in
the past, so this doens't always result in a new event being stored in
the DB.
- `email_analytics_events_stored` — a counter incremented each time an
event is stored in the DB. For example, if an email is opened 3 times by
the same recipient, this counter will only be incremented once.
- The metrics also have a label for the event type, so we can split out
opened events from delivered events. We can use the `rate()` function in
grafana to then get an `events ingested per second` metric, and compare
sites with/without the job queue enabled.
ref PLG-227
- added the correct order params for admin replies to ensure they are
sorted oldest to newest.
- hardcoded since we want to ensure it remains that way.
ref PLG-227
- Updated logic that allows Admin Users on comments to interact with
some endpoints from a specific admin-only route.
- It pulls 2 admin specific routes:
- 1. an admin specific 'browse' route that includes hidden comments that
would otherwise be hidden from regular users and members.
- 2. A specific replies route, that would also include hidden comments
- This was needed in order to get accurate pagination.
- Additionally, it wires up the routes via `message-handler` that deal
with the potential cors issues.
- Includes style updates
---------
Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
ref https://linear.app/ghost/issue/AP-598
We want to make sure that webhooks for internal integrations are fired even
when the custom integrations limit has been hit. In order to test this we've
had to update the fixtureManager to allow passing the integration type when
inserting fixture webhooks into the db.
ref https://linear.app/ghost/issue/AP-598
This is not a change in functionality.
The ActivityPub integration is an `internal` integration, because we want it to
be available regardless of the plan a Ghost(Pro) site is on. However the
webhooks service is not able to differentiate between webhooks for a custom
integration and an internal one.
Rather than disable webhooks entirely when the custom integrations limit
active, we want to allow webhooks for internal integrations to work. The first
step towards that is keeping the listener for the model events and have the
limiting happen in the WebhookTrigger which allows us to be more specific as to
which webhooks should be triggered or not.
ref PLG-227
- Wired up a new endpoint that would be able to paginate replies as an
admin user.
- The difference compared to the members-api endpoint is that this
includes hidden comments and includes the html string which is normally
hidden from non-admin users
ref https://linear.app/ghost/issue/AP-598
We want to make sure that webhooks for internal integrations are fired even
when the custom integrations limit has been hit. In order to test this we've
had to update the fixtureManager to allow passing the integration type when
inserting fixture webhooks into the db.
ref https://linear.app/ghost/issue/AP-598
This is not a change in functionality.
The ActivityPub integration is an `internal` integration, because we want it to
be available regardless of the plan a Ghost(Pro) site is on. However the
webhooks service is not able to differentiate between webhooks for a custom
integration and an internal one.
Rather than disable webhooks entirely when the custom integrations limit
active, we want to allow webhooks for internal integrations to work. The first
step towards that is keeping the listener for the model events and have the
limiting happen in the WebhookTrigger which allows us to be more specific as to
which webhooks should be triggered or not.
ref https://linear.app/ghost/issue/PLG-230
- the `inReplyTo` relationship was not being loaded for replies which meant the mapper never hit the code which adds `in_reply_to_snippet`
- moved all `in_reply_to` code behind the `commentImprovements` labs flag
- updated tests to correctly disable/enable the flag
- added test for browsing comments with replies-to-replies so `in_reply_to_snippet` is captured in the snapshot
no ref
Added Prometheus metrics for the job queue throughput and email analytics throughput. We'll likely keep these around as good metrics to keep an eye on, though for the moment their primary function is to establish a baseline for users w/o the job queue enabled so we can observe the full impact once switching it on.
ref
https://linear.app/ghost/issue/ENG-1769/improve-pool-utilization-metric
- Currently the connection pool metrics are all point in time metrics,
and with a scrape interval of 15s this doesn't tell us a whole lot about
what's happening in the pool.
- This commit adds a Summary metric to track the elapsed time each
transaction has to wait to acquire a connection from the pool, which
should be a good indication of contention in the pool.
- Also moved the call to `prometheusClient.instrumentKnex` to after `initCore` in the boot process, because the metric depends on event listeners on `knex.client.pool`, and the pool gets destroyed and recreated in `initCore`, which removes the listeners
no issue
- Since we decided to use the pushgateway instead of running a metrics
server, this removes the metrics server and its e2e tests
- We may reintroduce it later, but for now this is a simpler setup
no issues
- with gscan v4.46.0, we introduced a new rule for the custom fonts
feature
- this updates the valid theme zip file to make the theme tests work
with the new rules
no issue
- Currently the prometheus client is only initialized on boot if enabled
via config, but if it's required in other files (i.e. to create a custom
metric) it will be initialized then
- This commit explicitly checks if the prometheus client is enabled via
config before initializing it, thus preventing it from being initialized
when disabled
no ref
Stripe made changes (again) that causes our donation tests to fail. This
round we use an if statement to try to make it more inclusive of cases,
as I've seen them use an accordion button, card button, and no button
all in the past 12h.
no issue
Stripe recently updated their checkout page to use React with Framer
Motion for animations, causing our Playwright tests to intermittently
fail when attempting to click the “Card” payment button. The standard
Playwright `.click()` method was unable to interact with the button
reliably due to animation-related delays, where the button was present
in the DOM but not fully interactable according to Playwright’s strict
visibility checks.
Switching to `dispatchEvent('click')` directly fires the click event on
the button, bypassing Playwright’s visibility and interactability
checks. This ensures the test can proceed without waiting for animations
to fully complete, resolving the issue with the Stripe checkout flow.
ref https://linear.app/ghost/issue/ENG-1749
Batch sending tests were failing with MySQL fairly regularly. It appears
to be a race condition where the listener for the batch sending job
having completed is returning too early, causing the subsequent
Bookshelf data model refresh to happen too soon.
This is a fundamental flaw in the JobManager awaitCompletion handler
(and how the batch sending system interacts with it) as there's no way
to identify one batch from another - they all use the same name, and we
don't pass along any metadata.
ref DES-949
This adds custom fonts feature allowing users to select heading and body fonts for their themes from a curated list. This allows publishers to have more control over their brand, and allows themes to have a wider range of styles to appeal to different audiences.
Without custom fonts support, themes will continue to work as normal, but users won't be able to customize their typography. As for the official themes, all of them will support custom fonts.
---------
Co-authored-by: Aileen Booker <AileenCGN@gmail.com>
ref PLG-227
- Behind flags
- Changed Comments API for members and guests to not return hidden or
removed comments - with the only exception being if a hidden or removed
comment have published replies, in which case it will be greyed out as
per the previous version on the UI.
- Wired up a new admin API endpoint for comment to receive all comments.
It's on par with the members / guests endpoint, with the difference
being that it it shows hidden comment's content, where previously the
html property was nullified.
ref
https://linear.app/ghost/issue/ENG-1592/start-monitoring-connection-pool-utilization-in-ghost
- This commit adds prometheus metrics to the connection pool so we can
start to track connection pool utilization, number of pending acquires,
and also adds some basic SQL query summary metrics like queries per
minute and query duration percentiles.
- The connection pool has now been theorized to be a main constraint of
Ghost for some time, but it's been challenging to get actual visibility
into the state of the connection pool. With this change, we should be
able to directly observe, monitor and alert on the connection pool.
- Updated grafana version to fix a bug in the query editor that was
fixed in 8.3, even though this is a couple versions ahead of production
no issue
- We had reintroduced nodemon in
af0f26c75f (diff-bf18f8caf848e17b35e266db04bcaeaad05a3e5d069846615d2b1260482396e1)
for the docker setup, but it has since caused some issues with the `yarn
dev` script.
- In particular, it was causing a restart while migrations were running
in development, which left a migration lock on and prevented Ghost from
starting.
- This commit removes nodemon and replaces it with node --watch, which
we had been using in the past without issues.
refs https://linear.app/ghost/issue/AP-500
We've got a new @tryghost/activitypub package, which is gonna handle all
of the wiring between Ghost and ActivityPub. Currently that is just the
configuration of webhooks for the internal ActivityPub integration.
All this logic is run on the boot of Ghost, though notably in a
non-blocking manner, it's initialised as part of the background services
so it should not have an effect on the time to serving requests. having
said that - it needs to be defensive against errors, which is why the
entire network request is in a try/catch, as well as lookups for the
integration able to handle missing data.
Unit tests use an in-memory sqlite instance, which means we're testing a
full flow, ideally there would be a way to get the schema from Ghost for
this, but for now this is acceptable IMO.
refs https://linear.app/ghost/issue/AP-500
Rather than having to manually create an integration for communication
with the ActivityPub service, we are going to have an internal
integration which will then be used to handle webhooks between Ghost &
ActivityPub
The 'internal' type has been used to keep it out of the UI/API but
available for all Pro customers, which is necessary during the private
beta.
---------
Co-authored-by: Michael Barrett <mike@ghost.org>
