refs. https://linear.app/tryghost/issue/DES-122/bookmark-card-issues
This PR addresses the following content card related problems:
1. The design of the following cards are more self-contained so it makes
more sense to use `px` for their font-sizes and spacings so it looks the
same regardless of the theme. Of course themes still can override these
values.
Updated cards to use `px` for font sizing:
- audio
- bookmark
- file
- product
2. So far header and signup cards had been using `rem` for font-sizes
and some sizing. This commit updates these to use `em` instead so that
it's consistent with all other cards.
3. The favicon sometimes is not available for bookmark cards. This PR also
fixes that by providing a default favicon for these cases.
no issue
Bumps `Comments-UI` app version that contains a few changes:
- comments data is now lazy-loaded with API requests being deferred until the comments block is scrolled into view, saving up-front visitor data usage as well as reducing server-load for page views where the comments are never seen
- comments data is now fetched from `/members/api/comments/{post_id}/` rather than using the post_id in the `filter` param to enable cache bucketing and cache invalidation
- `created_at` timestamp has been dropped from the initial comments data request so the results can be cached, on pagination requests the timestamp has been improved to use the created_at data from the response so it remains consistent and can also be cached
- `order` param has been dropped from API requests as the API has been updated to include our default ordering
closes ENG-681
There's no need to provide an `order` param with every request in Comments-UI if the API has default ordering that matches our requirements. The order param makes logs more noisy/harder to read than they need to be so we want to get rid of it.
- modified comments API input serializer to add a default order param to the browse and replies endpoints when none is provided
- removed order param from the requests that Comments-UI makes
refs https://linear.app/tryghost/issue/ENG-676/
We want to make sure that we're not serving stale liked counts for
comments, which means we need to cache bust when they're liked/unliked
Unfortuantely this means we need to fetch the comment from the db so
that we have access to the post id.
refs https://linear.app/tryghost/issue/ENG-676/
This is the meat of the change and actually causes the cache to be
invalidated on adds and edits to the comments endpoints.
It doesn't currently include the liked/unliked actions at the moment
as we don't have easy access to the post id from those endpoints.
refs https://linear.app/tryghost/issue/ENG-676/
This is pretty simple as we can reuse the existing browse method
on the CommentsController, but we need to add support for the post_id
option to the endpoint, for it to be added to the frame.
We also need to update the browse method to enforce the post_id on the
NQL filter. I initially tried this with string concatenation, but ran
into way too many bugs, so we're using a mongo transformer instead.
refs https://linear.app/tryghost/issue/ENG-670
We keep running into issues with a sites content not being correct,
and slow get helpers being the suspect - but it's difficult to prove.
The idea behind this it to give us concrete evidence, which will allow
us to diagnose the problem faster.
closes ENG-660
- added tagged template function to strip leading whitespace from our plaintext email strings without making the source file harder to read
closes ENG-666
- the Admin API `GET /slugs/{type}/{slug}/` endpoint is used by Admin to check when a potential slug needs de-duping by adding a `-{x}` suffix. Most often this occurs when setting a draft post title
- the endpoint was returning a full-site cache invalidation header meaning hosting services could be blowing away their site caches and needlessly hurting performance because this endpoint is purely a read operation and makes no changes to the site
- updated the endpoint to return no cache invalidation header
closes ENG-666
- the Admin API `GET /slugs/{type}/{slug}/` endpoint is used by Admin to check when a potential slug needs de-duping by adding a `-{x}` suffix. Most often this occurs when setting a draft post title
- the endpoint was returning a full-site cache invalidation header meaning hosting services could be blowing away their site caches and needlessly hurting performance because this endpoint is purely a read operation and makes no changes to the site
- updated the endpoint to return no cache invalidation header
no refs
- Offers browser tests were subject to a race condition. I'm guessing
this dates back to when we moved to Settings X (and React), as it seems
the url for the offer is not present on the first render of the page -
despite being returned in the `POST` request of the offer creation, the
component does a `GET` on render to get the link. This is now awaited.
- The Publishing timezone test also seemed to suffer from a race
condition. This is less sure of a fix as it's a much less frequent
failure. The date time picker input is now validated in the test before
continuing.
- Offers browser tests often timed out so the timeout has been moved to
90s for these tests.
- All tests were bumped to 75s timeout as we generally would
occasionally hit the timeout.
closes ENG-608
- bumps Koenig rendering packages to include fix for HTML entities in HTML card content being decoded during rendering which could result in unexpected/broken output
ref ENG-607
- also added the option to show the monthly pricing by default during
signup
Co-authored-by: Simon Backx <simon@ghost.org>
Co-authored-by: Djordje Vlaisavljevic <dzvlais@gmail.com>
no issue
- to test if we can access Private Sites in Admin when set as a private
site.
- the issue is, we have CORS issues that doesn't allow a cookie to be
passed via Admin when the site uses a custom domain.
- generally does not affect self hosters.
refs https://linear.app/tryghost/issue/ENG-600
- users need an option so they can perform actions like delete users
without blowing up Ghost as large dbs can OOM node
refs INC-18
- release v5.69.0 introduced a data discrepancy in the free tier
visibility: the "free" tier visibility got out of sync with the
"portal_plans" setting due to a bug in the new Admin settings. The bug
was corrected in a patch release rolled out a few days later, v5.69.4
- however, the data discrepancy has not been corrected for all
customers; this data migration fixes the data discrepancy
refs https://linear.app/tryghost/issue/ENG-599
- Portal tests occasionally failed without clear cause on CI, possibly
due to GH runner region
- Portal tests never successfully ran locally for US-based IPs because
of a required prompt for Stripe Pass
refs https://linear.app/tryghost/issue/ENG-599
- member count is based on the cache which only updates ~every minute
- forced cache clear on manual member add/delete (not import)
- tests were failing based on the assumption that a new site that adds a
member has a nonzero member count, although the cache did not reflect
this quickly enough for the test to pass
Previously on a new site if you tried to publish a newsletter, it would
require at least one member. If you quickly added a member and tried to
send a newsletter, it would stop you saying you need at least one
member, requiring a browser refresh. This was a bug that is resolved
with this changes, as well as odd behaviour to try to write tests
around.
refs https://github.com/TryGhost/Toolbox/issues/501
- at this point, we have no real reason to keep this behind as it wasn't
proven what the cause of the high CPU was, and it's just causing more
lockfile issues with the resolution
closes https://github.com/TryGhost/Product/issues/4234
- bumps Koenig packages to version containing a fix to our denest transform so it properly handles denesting element nodes inside list item nodes
ref https://github.com/TryGhost/Ghost/issues/12802
fixes DMA-27
- You can choose any support and newsletter email address in the UI
without verification (as long as your SMTP-server / Mailgun can send
from it)
- All emails will use the mail.from config as the from address as a
default:
- Staff notification emails no longer use the made up ghost@domain email
address
- Newsletters no longer default to 'noreply@domain'
- Member related emails (signin/signup/comment notifications...) will
continue to be send from the chosen support address (Portal settings →
Account page), but will now default to the mail.from config instead of
noreply@domain if no support address is set.
refs ENG-599
- added refresh to publishing workflow test
- member count is cached and not updated immediately upon adding a
member, but a count >0 is required in order to send a newsletter (what
this test tests)
- we are looking at updating the cached count; until then, a refresh
will be a performance hit but allow this test to pass
no issue
- Renaming the configuration parameter created in this commit:
e0dae46dfc
- No functional difference, this change just makes the configuration a
bit more succinct
no issue
- To help debug potential causes of slow/aborted get helpers, it would
be cool to get more visibility into how Ghost handles database
connections, particularly if it has to spend a long time waiting to
acquire a new connection from the pool.
- Under the hood, knex uses a package called tarn
(https://github.com/Vincit/tarn.js/tree/3.0.2) to manage the connection
pool. Tarn provides some hooks for instrumentation, so we can use those
to get some basic visibility into the connection pool.
- This PR adds handling for creating, acquiring and releasing
connections from Tarn's connection pool which logs some basic metrics,
particularly the queue length and time it takes to acquire a connection.
no issue
- To help debug ABORTED_GET_HELPER errors, this PR adds Sentry
instrumentation to the get helpers
- It also adds the homepage, any pages/posts, the tag page, and the
author page to the list of transactions that will send to Sentry
refs TryGhost/Ghost#19559
- custom excerpts are truncated based on character length
- escaped characters added extra length but we didn't account for this,
resulting in poor truncation of excerpts
refs TryGhost/Ghost#19559
- custom excerpts are truncated based on character length
- escaped characters added extra length but we didn't account for this,
resulting in poor truncation of excerpts
fixes PROD-102
When a newsletter has a sender_email stored in the database that Ghost
is not allowed to send from, we no longer return it as sender_email in
the API. Instead we return it as the sender_reply_to. That way the
expected behaviour is shown correctly in the frontend and the API result
also makes more sense.
In addition to that, when a change is made to a newsletters reply_to
address we'll clear any invalid sender_email values in that newsletter.
That makes sure we can clear the sender_reply_to value instead of
keeping the current fallback to sender_email if that one is stored.
On top of that, this change correclty updates the browse endpoint to use
the newsletter service instead of directly using the model.
no issue
- we recently added a redirect to disable access to the preview endpoint for sent email-only posts but the condition was too broad and also disabled access to scheduled email-only posts
- adjusted so we only apply the /p/ -> /email/ redirect for sent posts
refs TryGhost/Product#4243
- Externally hosted images added in the editor were not populating the
`width` and `height` attributes, which could result in overflowing
images in certain email clients, particularly Outlook.
- This fix populates the `width` and `height` attributes in the editor
when adding an external image by URL or copy/pasting, which in turn
corrects the rendering in Outlook.
- Various other fixes and improvements to editor related packages, see
https://github.com/tryghost/koenig repo for more info
refs PROD-215 PROD-216
- Added toast notifications for successful sender and reply-to email
address change behind the flag, instead of the modal
- Updated email template for verifying new sender or reply-to email
closes https://github.com/TryGhost/Product/issues/4247
- bumps `@tryghost/kg-default-transforms` with a fix to our de-nesting transform so ListNode is no longer ignored as a badly nested child node which can occur through copy/paste from other editors
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
refs TryGhost/Product#4243
- Externally hosted images added in the editor were not populating the
`width` and `height` attributes, which could result in overflowing
images in certain email clients, particularly Outlook.
- This fix populates the `width` and `height` attributes in the editor
when adding an external image by URL or copy/pasting, which in turn
corrects the rendering in Outlook.
- Various other fixes and improvements to editor related packages, see
https://github.com/tryghost/koenig repo for more info
refs PROD-215 PROD-216
- Added toast notifications for successful sender and reply-to email
address change behind the flag, instead of the modal
- Updated email template for verifying new sender or reply-to email
no issue
- issue reported via the forum https://forum.ghost.org/t/video-embed-break-page-on-mobile/44172
- due to historical issues we check against http/https and non-www/www URLs to match an oembed provider in case our library's provider list is out of date. However we checked http first which could match and then update the original URL to be `http` in place of `https` leading to potentially broken oembed fetch requests as was the case with http://odysee.com URLs
refs https://linear.app/tryghost/issue/BIZ-6/[wip]-update-segment-events
- With the removal of the `integration.added` event, we have no more model events remaining to listen to for our analytics
- Removal of the function entirely seems the easier and more straightforward way
refs https://linear.app/tryghost/issue/BIZ-6/[wip]-update-segment-events
- Removed model events to listen to: `post.published`, `page.published`, and `theme.uploaded` in segment service, as we're not actively using those.
- Updated tests to reflect the changes (from 4 events to 1 model event)
This logic is so simple it isn't worth having the indirection of another class.
This also removes the indirection of wrapped getters/setters, which is useful
because otherwise we need to update the wrapper with new methods each time
theunderlying implementation is changed. There was a note about losing the
context of this, but I haven't found anywhere that the context is lost.
closes https://github.com/TryGhost/Product/issues/4247
- bumps `@tryghost/kg-default-transforms` with a fix to our de-nesting transform so ListNode is no longer ignored as a badly nested child node which can occur through copy/paste from other editors
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
The main changes are:
- Updating the pipeline to allow for doing a background refresh of the
cache
- Remove the use of the EventAwareCacheWrapper for the posts public
cache
### Background refresh
This is just an initial implementation, and tbh it doesn't sit right
with me that the logic for this is in the pipeline - I think this should
sit in the cache implementation itself, and then we call out to it with
something like: `cache.get(key, fetchData)` and then the updates can
happen internally.
The `cache-manager` project actually has a method like this called
`wrap` - but every time I've used it it hangs, and debugging was a pain,
so I don't really trust it.
### EventAwareCacheWrapper
This is such a small amount of logic, I don't think it's worth creating
an entire wrapper for it, at least not a class based one. I would be
happy to refactor this to use a `Proxy` too, so that we don't have to
add methods to it each time we wanna change the underlying cache
implementation.
no issue
When creating the batches when sending an email, we log a message to
Sentry when there is an unexpected offset of 1% between creating the
email and actually creating the batch recipients. We used a method that
was not mapped in our Sentry proxy.
Location of error: ghost/email-service/lib/BatchSendingService.js:286
- we want to pass in the schema tables instead of cross requiring them
from a different package because it means the package isn't standalone
and moving the code structure around breaks the data generator
ref PROD-233
- Stored whether Docker is used in the config files
- When running `yarn setup`, any existing Docker container will be
reset. Run with `-y` to skip the confirmation step.
- `yarn setup` will always init the database and generate fake data
- Increased amount of default generated data to 100,000 members + 500
posts.
- Made lots of performance improvements in the data generator so we can
generate the default data in ±170s
- we don't need to load this if we haven't configured Node profiling to occur
- this might help fix random segfaults we've been seeing in CI, which
only started occurring after this dependency was added
refs
[ARCH-33](https://linear.app/tryghost/issue/ARCH-33/fix-sentry-environment)
To ensure that we are correctly identifying the environment that data is
being sent to Sentry from, we can use the `PRO_ENV` environment variable
if it is available. This will be set to `production` in production and
`staging` in staging. If `PRO_ENV` is not available, we will fall back
to retrieving the environment from config (`env`)
refs ARCH-29
- Added Sentry Profiling to collect more detailed performance data on
the backend.
- This feature is opt-in behind a config. To enable profiling, first
enable tracing with `sentry.tracing.enabled: true`, then set
`sentry.profiling.enabled: true` and `sentry.profiling.sampleRate` to a
decimal number between 0 and 1.
refs ARCH-21
- We currently have NewRelic setup for a few of our largest customers
for monitoring performance, but it is too expensive to enable across all
sites
- Sentry has similar (but simpler) performance monitoring tools to keep
track of response times that are available to us for free, but we just
haven't configured them
- This PR sets up Sentry Performance monitoring for API requests so we
can have one place for monitoring errors + performance so we can stay on
top of response times more easily.
- Tracing is disabled by default, so there is no additional overhead
unless `sentry.tracing.enabled` is set to `true` in the site's config.
Additionally, `sentry.tracing.sampleRate` should be set to a decimal
value between 0 and 1. This value defaults to 0 to avoid accidentally
blowing through quota, and requires a value to explicitly be set in
order to send the traces to Sentry.
refs
[ARCH-33](https://linear.app/tryghost/issue/ARCH-33/fix-sentry-environment)
To ensure that we are correctly identifying the environment that data is
being sent to Sentry from, we can use the `PRO_ENV` environment variable
if it is available. This will be set to `production` in production and
`staging` in staging. If `PRO_ENV` is not available, we will fall back
to retrieving the environment from config (`env`)
fixes PROD-290
- in order to receive webmentions (e.g. recommendations), Ghost sites
expose a /webmentions/receive endpoint. This endpoint is wrongly being
indexed by Google as a regular page, and causes indexing errors in
Google Search Console
- we don't need to load this if we haven't configured Node profiling to occur
- this might help fix random segfaults we've been seeing in CI, which
only started occurring after this dependency was added
refs ARCH-29
- Added Sentry Profiling to collect more detailed performance data on
the backend.
- This feature is opt-in behind a config. To enable profiling, first
enable tracing with `sentry.tracing.enabled: true`, then set
`sentry.profiling.enabled: true` and `sentry.profiling.sampleRate` to a
decimal number between 0 and 1.
refs TryGhost/Product#4125
This PR adds two new integration tests to ensure all our Koenig cards
are rendered properly after going through the EmailRenderer. Although we
have thorough tests for the cards themselves in the Koenig repo, the
EmailRenderer does post-processing on the rendered HTML, such as
inlining CSS, which can adversely impact the rendered output of our
cards in email clients (usually Outlook).
Since email newsletters are a core feature of Ghost, these bugs are
typically fairly urgent, and since it is email, they are also quite
difficult to troubleshoot and fix. These two tests are intended to
prevent bugs of this sort, which in the past have been created by
seemingly harmless changes like bumping dependencies that are used in
the EmailRenderer.
The idea is to create a 'Golden Post' which has at least 1 of every card
from Koenig, run that post through the EmailRenderer, and take a
snapshot of the rendered HTML. In the future, if we make any changes to
the EmailRenderer or the Koenig cards themselves, this will trigger us
to carefully consider the changes, and it provides an 'expected' output
to compare our changes against.
Additionally, the second test simply checks that all cards from
`kg-default-nodes` are included in the 'Golden Post'. This protects
against any new cards that we will add in the future — as soon as we add
them to Koenig and bump `kg-default-nodes` in Ghost, this test will
fail, prompting us to add the new card to the Golden Post and update the
snapshots.
We should also run the 'Golden Post' through a test in Litmus, which
allows us to visually inspect the rendered email across many different
email clients. Ideally we would create a process to review the output of
the 'Golden Post' in Litmus whenever we update the snapshot as well.
refs ARCH-21
- We currently have NewRelic setup for a few of our largest customers
for monitoring performance, but it is too expensive to enable across all
sites
- Sentry has similar (but simpler) performance monitoring tools to keep
track of response times that are available to us for free, but we just
haven't configured them
- This PR sets up Sentry Performance monitoring for API requests so we
can have one place for monitoring errors + performance so we can stay on
top of response times more easily.
- Tracing is disabled by default, so there is no additional overhead
unless `sentry.tracing.enabled` is set to `true` in the site's config.
Additionally, `sentry.tracing.sampleRate` should be set to a decimal
value between 0 and 1. This value defaults to 0 to avoid accidentally
blowing through quota, and requires a value to explicitly be set in
order to send the traces to Sentry.
refs https://github.com/TryGhost/Product/issues/4209
- bumped Koenig packages
- `koenig-lexical` added nested editor TK support
- all packages dropped Node 16 support
- switched to using `isTKEnabled` prop and `<TKCountPlugin>`
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
fixes PROD-61
This adds a new default plan setting. It defaults to yearly, which is
the current default selected interval in Portal.
Behind the new portal improvements feature flag, the default plan can be
changed. It will also change automatically if the available intervals
are changed.
This PR also wires up passing the new setting to the Portal preview.
fixes https://github.com/TryGhost/Product/issues/4216
When generating page/post metadata, we generate a "context object" from
several meta helpers. In the event there is no context from the resource
type, we generate a fallback object.
However... we generate this fallback object no matter what.
Now, the fallback object is just 3x settingsCache.get, which should be
fast, but it's not. I've opened a separate issue for that: https://github.com/TryGhost/Product/issues/4217
In the mean time, we can switch this logic around to only do these calls
when we have no other context.
From testing, this allows for 10% more throughput on a post 🤯
fixes GRO-72
- added "default_email_address" and "support_email_address" to the
public settings
- when available, use these addresses in Portal. Otherwise, fallback to
current logic
refs https://github.com/TryGhost/Product/issues/4196
The offers API basically returns the data you pass to it, rather than
the created database record. It looks like this is how it was intended
to work in the first place; the `setMilliseconds` is because the test
helper expects `.000Z`, which I assume is because MySQL will strip off
the milliseconds when it's saved.
fixes GRO-71
- Current flow: unchanged
- New managed flow: verification required
- New managed flow with custom sending domain: only verification
required for different domains
- Self hosters (feature flag): no verification required
closes https://github.com/TryGhost/Product/issues/4191
Without this patch, themes can read arbitrary files from your system and
expose them to the internet via the layout feature of express-hbs.
For example `{{!< ../../../../config.production.json}}` would spit out config,
which can contain secrets.
As theme upload is restricted to users with the Admin role, this mostly effects
hosting providers which use their own secret keys for e.g. mail or database config
refs https://github.com/TryGhost/Product/issues/4181
We were seeing slow queries when joining on this table, and the index
speeds them up. The down migration is tricky because when we add the
index MySQL can optimise away some `KEY` indexes on the `newsletter_id`
column. When we then go to remove the newly created index, there is no
index for the FK!
We also remove the use of `force index` as 1. the index we're forcing is
optimised away and 2. we don't need it anymore!
Co-authored-by: Daniel Lockyer <hi@daniellockyer.com>
refs GRO-80
- added a new meta field "email_verified" to the /verification endpoint
for newsletters. This meta field contains which email has been verified,
"sender_email" or "sender_reply_to"
- updated copy in newsletter settings, based on which email has been
verified
fixes GRO-73
We need to avoid duplicating the complex logic for determining the
default email address and the support email address. So these are now
exposed as calculated settings.
ref GRO-54
fixes GRO-63
fixes GRO-62
fixes GRO-69
When the config `hostSettings:managedEmail:enabled` is enabled, or the
new flag (`newEmailAddresses`) is enabled for self-hosters, we'll start
to check the from addresses of all outgoing emails more strictly.
- Current flow: nothing changes if the managedEmail config is not set or
the `newEmailAddresses` feature flag is not set
- When managedEmail is enabled: never allow to send an email from any
chosen email. We always use `mail.from` for all outgoing emails. Custom
addresses should be set as replyTo instead. Changing the newsletter
sender_email is not allowed anymore (and ignored if it is set).
- When managedEmail is enabled with a custom sending domain: if a from
address doesn't match the sending domain, we'll default to mail.from and
use the original as a replyTo if appropriate and only when no other
replyTo was set. A newsletter sender email addresss can only be set to
an email address on this domain.
- When `newEmailAddresses` is enabled: self hosters are free to set all
email addresses to whatever they want, without verification. In addition
to that, we stop making up our own email addresses and send from
`mail.from` by default instead of generating a `noreply`+ `@` +
`sitedomain.com` address
A more in depth example of all cases can be seen in
`ghost/core/test/integration/services/email-addresses.test.js`
Includes lots of new E2E tests for most new situations. Apart from that,
all email snapshots are changed because the from and replyTo addresses
are now included in snapshots (so we can see unexpected changes in the
future).
Dropped test coverage requirement, because tests were failing coverage
locally, but not in CI
Fixed settings test that set the site title to an array - bug tracked in
GRO-68
refs TryGhost/Product#4175
- Added error handling to Sentry's beforeSend function in both Admin and
Core, so if there is any error in beforeSend, we will still send the
unmodified event to Sentry
- This is in response to an incident yesterday wherein the beforeSend
function threw an error due to an unexpected missing value in the
exception. The event sent to Sentry was the error in the beforeSend
function, and the original error never reached Sentry.
- If the original event had reached Sentry, even if unmodified by the
logic in beforeSend, we could have been alerted to the issue sooner and
more easily identified all affected sites.
- Also added defensive logic to protect for certain values in the
exception passed to beforeSend not existing and added unit tests for the
beforeSend function in admin and core
refs https://github.com/TryGhost/Product/issues/4181
- we've seen MySQL change its query planner to use a different index
than the ideal one, resulting in drastically slower query performance
when fetching newsletters with the member count
- this forces the use of the ideal index on MySQL
- this kept many of the Ghost team up overnight, so I hope MySQL is
happy
