no ref
Added Prometheus metrics for the job queue throughput and email analytics throughput. We'll likely keep these around as good metrics to keep an eye on, though for the moment their primary function is to establish a baseline for users w/o the job queue enabled so we can observe the full impact once switching it on.
no ref
- [x] Created a new folder `ne` based on locale code for Nepali locale
inside `ghost/i18n/locales/`
- [x] Cloned 6 new JSON files pertaining to `en` folder from English
locale.
- [x] Added all field translations from English to Nepali for
`ghost.json` file.
- [x] Added all field translations from English to Nepali for
`search.json` file.
- [x] Added all field translations from English to Nepali for
`signup-form.json` file.
- [ ] TODO: Remaining 3 other files viz. `comments`, `newsletter`, and
`portal`.
"Thank you for creating Ghost and making it open source.": "Ghost
सिर्जना गरेर यसलाई ओपन सोर्स बनाउनु भएकोमा धन्यवाद।"
no ref
Suggestion to update the translation for 'A label for the thumbs-up response in member feedback at the bottom of emails' to 'I like it' and 'I don't like it,' as this phrasing sounds better and is more appropriate in Brazilian Portuguese. Thank you.
ref
https://linear.app/ghost/issue/ENG-1769/improve-pool-utilization-metric
- Currently the connection pool metrics are all point in time metrics,
and with a scrape interval of 15s this doesn't tell us a whole lot about
what's happening in the pool.
- This commit adds a Summary metric to track the elapsed time each
transaction has to wait to acquire a connection from the pool, which
should be a good indication of contention in the pool.
- Also moved the call to `prometheusClient.instrumentKnex` to after `initCore` in the boot process, because the metric depends on event listeners on `knex.client.pool`, and the pool gets destroyed and recreated in `initCore`, which removes the listeners
no issue
- Since we decided to use the pushgateway instead of running a metrics
server, this removes the metrics server and its e2e tests
- We may reintroduce it later, but for now this is a simpler setup
no issue
- The `instrumentKnex` method was directly accessing the `promClient`
instance to create custom metrics, and keeping track of them manually in
a `customMetrics` map. This isn't necessary, since the metrics are all
tracked within the `promClient` instance's registry. This method now
uses the `prometheusClient.register...()` methods to create the metrics,
and retrieves them with the `getMetric()` method to reduce duplication
of work and manual bookkeeping
- This also removes the query count metric, as there is a count already
included in the query duration Summary metric
no issues
- with gscan v4.46.0, we introduced a new rule for the custom fonts
feature
- this updates the valid theme zip file to make the theme tests work
with the new rules
no issue
- Currently the prometheus client is only initialized on boot if enabled
via config, but if it's required in other files (i.e. to create a custom
metric) it will be initialized then
- This commit explicitly checks if the prometheus client is enabled via
config before initializing it, thus preventing it from being initialized
when disabled
ref
https://linear.app/ghost/issue/ENG-1771/add-utility-functions-to-easily-create-custom-metrics
- Currently adding custom metrics to our prometheus client requires you
to directly access the `prometheusClient.client` to create the metrics
- This isn't super convenient, as you then have to either keep the
metric in a local variable, or manually get it from the
`prometheusClient.client.register`
- This commit exposes some utility functions for registering metrics on
the `prometheusClient` class, and for retrieving metrics that have
already been registered
no ref
Stripe made changes (again) that causes our donation tests to fail. This
round we use an if statement to try to make it more inclusive of cases,
as I've seen them use an accordion button, card button, and no button
all in the past 12h.
no issue
Stripe recently updated their checkout page to use React with Framer
Motion for animations, causing our Playwright tests to intermittently
fail when attempting to click the “Card” payment button. The standard
Playwright `.click()` method was unable to interact with the button
reliably due to animation-related delays, where the button was present
in the DOM but not fully interactable according to Playwright’s strict
visibility checks.
Switching to `dispatchEvent('click')` directly fires the click event on
the button, bypassing Playwright’s visibility and interactability
checks. This ensures the test can proceed without waiting for animations
to fully complete, resolving the issue with the Stripe checkout flow.
ref https://linear.app/ghost/issue/ENG-1749
Batch sending tests were failing with MySQL fairly regularly. It appears
to be a race condition where the listener for the batch sending job
having completed is returning too early, causing the subsequent
Bookshelf data model refresh to happen too soon.
This is a fundamental flaw in the JobManager awaitCompletion handler
(and how the batch sending system interacts with it) as there's no way
to identify one batch from another - they all use the same name, and we
don't pass along any metadata.
ref DES-949
This adds custom fonts feature allowing users to select heading and body fonts for their themes from a curated list. This allows publishers to have more control over their brand, and allows themes to have a wider range of styles to appeal to different audiences.
Without custom fonts support, themes will continue to work as normal, but users won't be able to customize their typography. As for the official themes, all of them will support custom fonts.
---------
Co-authored-by: Aileen Booker <AileenCGN@gmail.com>
ref PLG-227
- Behind flags
- Changed Comments API for members and guests to not return hidden or
removed comments - with the only exception being if a hidden or removed
comment have published replies, in which case it will be greyed out as
per the previous version on the UI.
- Wired up a new admin API endpoint for comment to receive all comments.
It's on par with the members / guests endpoint, with the difference
being that it it shows hidden comment's content, where previously the
html property was nullified.
ref https://ghost.org/changelog/internationalization/
- I've changed "email" to "e-mail", which is more correct in French. The
official term is "courriel" but it sounds a bit old-fashioned.
- Apart from that, I've made a few harmonisations and small corrections to
ensure consistency between the files.
ref
https://linear.app/ghost/issue/ENG-1592/start-monitoring-connection-pool-utilization-in-ghost
- This commit adds prometheus metrics to the connection pool so we can
start to track connection pool utilization, number of pending acquires,
and also adds some basic SQL query summary metrics like queries per
minute and query duration percentiles.
- The connection pool has now been theorized to be a main constraint of
Ghost for some time, but it's been challenging to get actual visibility
into the state of the connection pool. With this change, we should be
able to directly observe, monitor and alert on the connection pool.
- Updated grafana version to fix a bug in the query editor that was
fixed in 8.3, even though this is a couple versions ahead of production
ref https://github.com/TryGhost/Ghost/pull/21573
- We discussed this, and figured we should get this behind the flag for
now, even though it's pulled
- This gives us the ability to do a fast revert if we have to later,
which is the point of having the labs "GA FEATURES" array
- Note this removes the labsEnabled variable in the tests to make it
clearer if the flag is off or on (it was off!)
ref https://github.com/TryGhost/Ghost/pull/21573
- We discussed this, and figured we should get this behind the flag for
now, even though it's pulled
- This gives us the ability to do a fast revert if we have to later,
which is the point of having the labs "GA FEATURES" array
- Note this removes the labsEnabled variable in the tests to make it
clearer if the flag is off or on (it was off!)
ref https://github.com/moment/luxon/blob/master/docs/intl.md
- We noticed the following error trace: RangeError: Incorrect locale
information provided
at BatchSendingService.retryDb
(/home/ghost/node_modules/@tryghost/email-service/lib/BatchSendingService.js:639:32)
at new DateTimeFormat (<anonymous>)
at getCachedDTF
(/home/ghost/node_modules/luxon/build/node/luxon.js:621:11)
at new PolyDateFormatter
(/home/ghost/node_modules/luxon/build/node/luxon.js:842:16)
at Locale.dtFormatter
(/home/ghost/node_modules/luxon/build/node/luxon.js:1066:12)
at Formatter.dtFormatter
(/home/ghost/node_modules/luxon/build/node/luxon.js:2274:21)
at Formatter.formatDateTime
(/home/ghost/node_modules/luxon/build/node/luxon.js:2280:17)
at DateTime.toLocaleString
(/home/ghost/node_modules/luxon/build/node/luxon.js:6893:78)
at formatDateLong
(/home/ghost/node_modules/@tryghost/email-service/lib/EmailRenderer.js:45:74)
at Object.getValue
(/home/ghost/node_modules/@tryghost/email-service/lib/EmailRenderer.js:683:47)
at
/home/ghost/node_modules/@tryghost/email-service/lib/SendingService.js:158:36
at Array.map (<anonymous>)
at
/home/ghost/node_modules/@tryghost/email-service/lib/SendingService.js:154:54
at Array.map (<anonymous>)
at SendingService.buildRecipients
(/home/ghost/node_modules/@tryghost/email-service/lib/SendingService.js:151:24)
at SendingService.send
(/home/ghost/node_modules/@tryghost/email-service/lib/SendingService.js:127:33)
at response.retryDb
(/home/ghost/node_modules/@tryghost/email-service/lib/BatchSendingService.js:451:51)
- This is due to the locale being user-input - it can be set to any
string.
- In our email sending code we pass the string to luxon to format dates,
which errors if the locale is not valid according it Intl.
- This fix ensures that the locale is valid before passing it to luxon,
falling back to en-gb if the locale is not valid
ref https://github.com/moment/luxon/blob/master/docs/intl.md
- We noticed the following error trace: RangeError: Incorrect locale
information provided
at BatchSendingService.retryDb
(/home/ghost/node_modules/@tryghost/email-service/lib/BatchSendingService.js:639:32)
at new DateTimeFormat (<anonymous>)
at getCachedDTF
(/home/ghost/node_modules/luxon/build/node/luxon.js:621:11)
at new PolyDateFormatter
(/home/ghost/node_modules/luxon/build/node/luxon.js:842:16)
at Locale.dtFormatter
(/home/ghost/node_modules/luxon/build/node/luxon.js:1066:12)
at Formatter.dtFormatter
(/home/ghost/node_modules/luxon/build/node/luxon.js:2274:21)
at Formatter.formatDateTime
(/home/ghost/node_modules/luxon/build/node/luxon.js:2280:17)
at DateTime.toLocaleString
(/home/ghost/node_modules/luxon/build/node/luxon.js:6893:78)
at formatDateLong
(/home/ghost/node_modules/@tryghost/email-service/lib/EmailRenderer.js:45:74)
at Object.getValue
(/home/ghost/node_modules/@tryghost/email-service/lib/EmailRenderer.js:683:47)
at
/home/ghost/node_modules/@tryghost/email-service/lib/SendingService.js:158:36
at Array.map (<anonymous>)
at
/home/ghost/node_modules/@tryghost/email-service/lib/SendingService.js:154:54
at Array.map (<anonymous>)
at SendingService.buildRecipients
(/home/ghost/node_modules/@tryghost/email-service/lib/SendingService.js:151:24)
at SendingService.send
(/home/ghost/node_modules/@tryghost/email-service/lib/SendingService.js:127:33)
at response.retryDb
(/home/ghost/node_modules/@tryghost/email-service/lib/BatchSendingService.js:451:51)
- This is due to the locale being user-input - it can be set to any
string.
- In our email sending code we pass the string to luxon to format dates,
which errors if the locale is not valid according it Intl.
- This fix ensures that the locale is valid before passing it to luxon,
falling back to en-gb if the locale is not valid
no issue
- We had reintroduced nodemon in
af0f26c75f (diff-bf18f8caf848e17b35e266db04bcaeaad05a3e5d069846615d2b1260482396e1)
for the docker setup, but it has since caused some issues with the `yarn
dev` script.
- In particular, it was causing a restart while migrations were running
in development, which left a migration lock on and prevented Ghost from
starting.
- This commit removes nodemon and replaces it with node --watch, which
we had been using in the past without issues.
ref https://ghost.org/changelog/internationalization/
- It's difficult out of context to know whether to use an infinitive,
_Ajouter votre commentaire_ or an imperative, _Ajoutez votre
commentaire_. English doesn't make the difference, but French does. By
default, an infinitive is preferable because it remains a suggestion,
whereas an imperative gives an order. For example, I think that _Ajoutez
votre commentaire_ is not relevant, because you don't necessarily have
to add a comment, but you can, so you should say _Ajouter votre
commentaire_. I've tried to be as precise as possible, but you may need
to make some adjustments with the context.
- I find that the word _membre_ is too literal a translation of the
English language and includes too many innuendos such as "member of a
community", "member of a clan", "member of a club", and frankly, it's
not relevant to many blogs. The word _abonné_ is more relevant, because
it doesn't imply a social affiliation. As for the discussion, I've even
taken the liberty of writing _discussion entre lecteurs_, because that's
essentially what it is. It's not a discussion between members or
subscribers, it's a discussion between _readers_.
- Some placeholders are completely American (Jamie Larson), so I've
replaced him with a French one (Jean Martin).
- Complimentary is a word that I didn't understand in the context of
Ghost. I have the impression that it means _offert_, but I don't
understand when it's used.
refs https://linear.app/ghost/issue/AP-500
We've got a new @tryghost/activitypub package, which is gonna handle all
of the wiring between Ghost and ActivityPub. Currently that is just the
configuration of webhooks for the internal ActivityPub integration.
All this logic is run on the boot of Ghost, though notably in a
non-blocking manner, it's initialised as part of the background services
so it should not have an effect on the time to serving requests. having
said that - it needs to be defensive against errors, which is why the
entire network request is in a try/catch, as well as lookups for the
integration able to handle missing data.
Unit tests use an in-memory sqlite instance, which means we're testing a
full flow, ideally there would be a way to get the schema from Ghost for
this, but for now this is acceptable IMO.
refs https://linear.app/ghost/issue/AP-500
Rather than having to manually create an integration for communication
with the ActivityPub service, we are going to have an internal
integration which will then be used to handle webhooks between Ghost &
ActivityPub
The 'internal' type has been used to keep it out of the UI/API but
available for all Pro customers, which is necessary during the private
beta.
---------
Co-authored-by: Michael Barrett <mike@ghost.org>
ref https://ghost.org/changelog/internationalization/
- this fixes an error in which changing the language (or sliding the i18n
toggle) didn't correctly update email-service.
---------
Co-authored-by: Sag <guptazy@gmail.com>
ref https://linear.app/tryghost/issue/PLG-230
- adds `in_reply_to_id` to API output
- adds `in_reply_to_snippet` to API output
- dynamically generated from the HTML of the replied-to comment
- excluded if the replied-to comment has been deleted or hidden
- adds `commentSnippet` to `@tryghost/html-to-plaintext`
- skips anchor tag URLs as they won't be useful for snippet purposes
- skips blockquotes so the snippet is more likely to contain the unique content of the replied-to comment when it's quoting a previous comment
- returns a single line (no newline chars)
- allows setting `in_reply_to_id` when creating comments
- id must reference a reply with the same parent
- id must reference a published comment
- adds email notification for the original reply author when their comment is replied to
ref https://linear.app/tryghost/issue/PLG-230
- `comments.in_reply_to_id` will be used to keep a reference to the comment that the new comment was directed at
- used only for replies-to-replies, will be `null` for the top-level parent and `null` for any replies directly to that parent
- technically allows for infinite nesting within a parent comment thread but we won't be using that ability for now
- `comments.parent_id` will be kept as it provides a useful optimisation for loading the top-level comments list
- we're not using `comments.parent_id` for this to keep complexity down and avoid the need for recursive lookups
refs https://linear.app/ghost/issue/AP-500
The logic for generating identity tokens, whilst small, is something
that we don't want to duplicate - as it concerns security & access - so
can easily break interactions between different services. We're gonna
need to use identity tokens as part of the initialisation of the
activitypub service, so this is pulling it out preemptively for that use
case
We shouldn't have logic inside of the endpoint controllers anyway, so
this is kinda general cleanup.
ref
https://linear.app/ghost/issue/ENG-1746/enable-ghost-to-push-metrics-to-a-pushgateway
- Trying to get Ghost working with the prometheus pushgateway in
staging, but it's logging an error each time it tries to push the
metrics. The error output is pretty useless for debugging, so this
commit improves the error messages to make it easier to debug.
no issue
- filled some missed translations but there are yet some missed
translations to complete. Actually i'm not managing newsletters so i
need more context to translate them correctly.
ref
https://linear.app/ghost/issue/ENG-1746/enable-ghost-to-push-metrics-to-a-pushgateway
- We'd like to use prometheus to expose metrics from Ghost, but the
"standard" approach of having prometheus scrape the `/metrics` endpoint
adds some complexity and additional challenges on Pro.
- A suggested simpler alternative is to use a pushgateway, to have Ghost
_push_ metrics to prometheus, rather than have prometheus scrape the
running instances.
- This PR introduces this functionality behind a configuration.
- It also includes a refactor to the current metrics-server
implementation so all the related code for prometheus is colocated, and
the configuration is a bit more organized. `@tryghost/metrics-server`
has been renamed to `@tryghost/prometheus-metrics`, and it now includes
the metrics server and prometheus-client code itself (including the
pushgateway code)
- To enable the prometheus client alone, `prometheus:enabled` must be
true. This will _not_ enable the metrics server or the pushgateway — it
will essentially collect the metrics, but not do anything with them.
- To enable the metrics server, set `prometheus:metrics_server:enabled`
to true. You can also configure the host and port that the metrics
server should export the `/metrics` endpoint on in the
`prometheus:metrics_server` block.
- To enable the pushgateway, set `prometheus:pushgateway:enabled` to
true. You can also configure the pushgateway's `url`, the `interval` it
should push metrics in (in milliseconds) and the `jobName` in the
`prometheus:pushgateway` block.
- using lodash to do this is unnecessarily heavy, so this commit
switches the code to the equivalent native version
- as mentioned in the comment I added, I think we can further optimize
this by storing it as a Set and then calling `Array.from` once, but
that's a step too far for now
- we can skip an extra `toJSON` call here by storing the value and
re-using it, which means we can cut down on the number of calls, which
can be particularly heavy when this codepath is hit hard
no issue
- following pull requests #21251 and #21497, this commit adds pt-BR
translations to newsletter.json and fixes other strings in pt-BR locale
files
refs https://github.com/TryGhost/Ghost/issues/19839
refs https://www.sqlite.org/limits.html
SQLite has limit of 500 items in a compound select statement.
This limit could be hit when a complex select statement was being
generated as part of a batch insert statement.
Lowering the batch size will have minimal impact on migration
performance while improving SQLite compatibility.
One of these bulk inserts is confirmed to be affected through the linked
issue.
I didn't confirm if the other two cases would trigger it, but the change
won't hurt there either.
ref https://linear.app/tryghost/issue/ENG-1556/
- added background job queue behind config flags
- when enabled, is only used for the member email analytics updates in
order to speed up the parent job, and take load off of the main process
that is serving requests
The intent here is to decouple certain code paths from the main process where it is unnecessary, or worse, where it's part of the request. Primary use cases are email analytics (particularly the member stats [open rate]) which are not particularly helpful in the period immediately following an email send, while the click traffic and delivered/opened events are.
Related, the email link clicks themselves send off a cascade of events that are quite a burden on the main process currently and are somewhat tied to the request response when they needn't be. We'll be looking to tackle that after some initial testing with the email analytics job.
ref PLG-220
- Added an `orderAttributes` override method to be able to pass
`count__likes` to the `findPage` DB helper.
- Unknowingly, without that override method in the model, it would strip
all 'non-default' queries.
- Adding that means we could remove our custom database queries and use
the regular `findPage` helper that also handles pagination.
no issue
This change adds the ability to pass a desired language/locale into
portal, as a data-locale tag added to the script that loads it.
Falls back to the previous behavior, loading from the site settings, if
no data-locale is present. (And then to 'en', if necessary.) No change
in expected behavior if data-locale is not passed.
This provides an option to run portal in a language other than the
sitewide language.
(ref: https://app.slack.com/client/T025584C4/C018EKC56JF)
- When we added i18n for search we missed gating it behind the i18n flag.
- There aren't that many translations for search yet, so it's likely not many have noticed yet
- We'll remove the flag soon, but until then, adding the flag for consistency :)
no ref
{{ghost_head}} is huge, and some power-users and theme creators want the
ability to customize what it contains. This PR makes it easier for a
theme to write custom schema, or to load a custom version of
portal/comments/search/etc, or to minimize load times by not loading
scripts where they aren't needed, in a theme-specific way.
Because ghost_head is controlled at the theme level, this gives folks in
managed hosting the new ability to load a different version of the
included app scripts (by preventing ghost_head from writing them and
adding them in manually).
Usage example: ` {{ghost_head exclude="search,portal"}} `
(empty array)
No changes to current behavior
search
The built-in sodo-search script
Includes adding the click event listener on buttons, generating the
search index, and the UI.
portal
The portal script
Handles sign-in and sign-up, payments, tips, memberships, etc, and all
the portal data-attributes.
announcement
The announcement bar javascript
If you'd like to use the announcement bar admin settings but not have it
[mess up your CLS
metric](https://www.spectralwebservices.com/blog/announcement-bar-a-review/),
this is for you.
metadata
Skips HTML tags for meta description, favicon, canonical url, robots,
referrer
Important for SEO
schema
The LD+JSON schema
Important for SEO
card_assets
Loads cards.min.css and .js
Needed on any page with a post body, unless your theme replaces them
all. Assets can also be selectively loaded with the [card_assets
override](https://ghost.org/docs/themes/content/?ref=spectralwebservices.com#editor-cards)
comment_counts
Loads the comment_counts helper
Needed if the page is using {{comments}} or data-ghost-comment-count
attribute
social_data
Produces the og: and twitter: attributes for social media sharing and
previews
Required for good social media cards
cta_styles
Removes the call to action (CTA) styles
Used for member signup and CTA cards - may be overwritten by your theme
already
ref https://linear.app/ghost/issue/ONC-506
- adding `context` with the returned API response makes the logged error much more useful as without it we only log the status code which misses any details for why the failure occurred
no issue
This PR adds the ability to translate the strings that appear in the
newsletter as boilerplate text, using i18next.
Variables are in single mustaches ( `{date}` ) in the translation
strings (rather than `{{date}}`), because these strings occur both the
email template.hbs and also .js files. That necessitated a separate
namespace.
This PR also includes changes to the newsletter button ("more like
this", "less like this", "comment") that were previously delivered on
desktop as images that included the text. @sanne-san provided a rework
that removed text-as-image from the desktop buttons, and allows more
shared code between the two layouts, along with making the buttons
translatable.
Example usage - handlebars
```
<h3 class="latest-posts-header">{{t 'Keep reading'}}</h3>
{{{t 'By {authors}' authors=post.authors }}}
```
(NOTE: triple { required because of possible & )
Example usage - javascript
```
getValue: (member) => {
if (member.status === 'comped') {
return t('complimentary');
}
if (this.isMemberTrialing(member)) {
return t('trialing');
}
// other possible statuses: t('free'), t('paid') //
return t(member.status);
}
```
---------
Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
Co-authored-by: Steve Larson <9larsons@gmail.com>
fix https://linear.app/ghost/issue/ENG-1105/settingscacheget-is-slow
- through profiling and flamegraphs, we can see that `_doGet` is one of
the bottlenecks during high traffic times, sometimes taking up to 20%
of the CPU time when hammering Ghost with `wrk`
- this is because, for the majority of settings cache lookup, we're
running `JSON.parse`, which blocks the main thread
- whilst we're only parsing small strings, we're doing it a LOT,
sometimes hundreds of times per request, which adds up
- this code just throws most deserializing at `JSON.parse`, so if we can
stop it from doing that, it'd be a huge win
- my initial attempts here were to convert the _doGet function to a
smarter deserializing, by looking up `cacheEntry.type` and acting
accordingly
- however, it became a bit of a logical nightmare, and difficult to
reason about for now (i still think we should do it)
- therefore, I'm just doing to add a hotpath fix to catch 99% of
usecases, which is checking the type of the cache entry and returning
the value if it's a string
- on a trivial benchmark locally, this causes Ghost to return 30% more
requests per second!!
ref PLG-220
- Improved `getBestComments` service to paginate correctly since we're
using a custom query to determine the top comments that goes beyond the
scope of what `findPage` is capable of.
- Updated CommentsController and CommentsService to support custom order
parameters.
- Added tests
no issue
- filtering was previously added to breadcrumbs but that wasn't enough to clean up Sentry reports
- added filtering to the `beforeSend` hook too so reports don't get cluttered with unhelpful XHR noise
ref https://app.incident.io/ghost/incidents/117
- the authenticate call made as part of signup was missed as part of the update when we adjusted the params for `cookie` authenticator's `authenticate` method in Admin so it could switch behaviour for 2fa
- fixed the authenticate call params and updated our mocked `/session` endpoint to check for expected POST data which would have let tests catch this error
