closes https://github.com/TryGhost/Product/issues/4247
- bumps `@tryghost/kg-default-transforms` with a fix to our de-nesting transform so ListNode is no longer ignored as a badly nested child node which can occur through copy/paste from other editors
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
fixes PROD-201
The issue was caused because we were searching the 'name' field instead of the 'title' field.
This also increases the performance when loading the posts:
- Makes sure no relations are loaded
- Only return the fields we actually need
- Stop using limit=all, and replaced it with network based search
refs https://github.com/TryGhost/Ghost/pull/19284
- previous commit had only partially rolled back usage of `errorInfo` variable but it wasn't caught because the line in question has linting disabled
The main changes are:
- Updating the pipeline to allow for doing a background refresh of the
cache
- Remove the use of the EventAwareCacheWrapper for the posts public
cache
### Background refresh
This is just an initial implementation, and tbh it doesn't sit right
with me that the logic for this is in the pipeline - I think this should
sit in the cache implementation itself, and then we call out to it with
something like: `cache.get(key, fetchData)` and then the updates can
happen internally.
The `cache-manager` project actually has a method like this called
`wrap` - but every time I've used it it hangs, and debugging was a pain,
so I don't really trust it.
### EventAwareCacheWrapper
This is such a small amount of logic, I don't think it's worth creating
an entire wrapper for it, at least not a class based one. I would be
happy to refactor this to use a `Proxy` too, so that we don't have to
add methods to it each time we wanna change the underlying cache
implementation.
no issue
The data generator created an offer for the free product. This caused an
error in admin UI because it couldn't find the tier for the offer.
This fixes the issue in both the data generator and the admin UI.
no issue
The data generator went out of memory when trying to generate fake data
for > 2M members. This adds some improvements to make sure it doesn't go
out of memory.
---------
Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
no issue
When creating the batches when sending an email, we log a message to
Sentry when there is an unexpected offset of 1% between creating the
email and actually creating the batch recipients. We used a method that
was not mapped in our Sentry proxy.
Location of error: ghost/email-service/lib/BatchSendingService.js:286
no issue
The members stats API is a lot more slower than the normal members count
cache, and we use the members count cache in a lot more places. So we
can cache it more.
no issue
When we open the editor, we fire 4 requests to fetch member counts. This
commit fixes this by replacing those calls with the members count cache
service.
no issue
- the members API endpoint by default adds `order by created_at` to the SQL queries which creates unnecessary overhead when we only care about a count because MySQL sorts the table before querying the single member
- specifying an explicit `order by id` overrides the default API behaviour
- locally with 2 million members the query times drop from >5sec to ~1sec
no refs
- member_count endpoint was queried multiple times on load
- moved to collectively use the member stats service
- prevented multiple tasks from being queued up to return count
- we want to pass in the schema tables instead of cross requiring them
from a different package because it means the package isn't standalone
and moving the code structure around breaks the data generator
ref PROD-233
- Stored whether Docker is used in the config files
- When running `yarn setup`, any existing Docker container will be
reset. Run with `-y` to skip the confirmation step.
- `yarn setup` will always init the database and generate fake data
- Increased amount of default generated data to 100,000 members + 500
posts.
- Made lots of performance improvements in the data generator so we can
generate the default data in ±170s
- we don't need to load this if we haven't configured Node profiling to occur
- this might help fix random segfaults we've been seeing in CI, which
only started occurring after this dependency was added
refs
[ARCH-33](https://linear.app/tryghost/issue/ARCH-33/fix-sentry-environment)
To ensure that we are correctly identifying the environment that data is
being sent to Sentry from, we can use the `PRO_ENV` environment variable
if it is available. This will be set to `production` in production and
`staging` in staging. If `PRO_ENV` is not available, we will fall back
to retrieving the environment from config (`env`)
refs ARCH-29
- Added Sentry Profiling to collect more detailed performance data on
the backend.
- This feature is opt-in behind a config. To enable profiling, first
enable tracing with `sentry.tracing.enabled: true`, then set
`sentry.profiling.enabled: true` and `sentry.profiling.sampleRate` to a
decimal number between 0 and 1.
refs ARCH-21
- We currently have NewRelic setup for a few of our largest customers
for monitoring performance, but it is too expensive to enable across all
sites
- Sentry has similar (but simpler) performance monitoring tools to keep
track of response times that are available to us for free, but we just
haven't configured them
- This PR sets up Sentry Performance monitoring for API requests so we
can have one place for monitoring errors + performance so we can stay on
top of response times more easily.
- Tracing is disabled by default, so there is no additional overhead
unless `sentry.tracing.enabled` is set to `true` in the site's config.
Additionally, `sentry.tracing.sampleRate` should be set to a decimal
value between 0 and 1. This value defaults to 0 to avoid accidentally
blowing through quota, and requires a value to explicitly be set in
order to send the traces to Sentry.
refs
[ARCH-33](https://linear.app/tryghost/issue/ARCH-33/fix-sentry-environment)
To ensure that we are correctly identifying the environment that data is
being sent to Sentry from, we can use the `PRO_ENV` environment variable
if it is available. This will be set to `production` in production and
`staging` in staging. If `PRO_ENV` is not available, we will fall back
to retrieving the environment from config (`env`)
fixes PROD-290
- in order to receive webmentions (e.g. recommendations), Ghost sites
expose a /webmentions/receive endpoint. This endpoint is wrongly being
indexed by Google as a regular page, and causes indexing errors in
Google Search Console
- we don't need to load this if we haven't configured Node profiling to occur
- this might help fix random segfaults we've been seeing in CI, which
only started occurring after this dependency was added
ref PROD-244
- Added support for canceled subscriptions and different subscription statusses
- Removed generation of susbcriptions table (not used)
- Added old canceled subscriptions for free members
- Added both positive and negative MRR events
ref PROD-244
Instead of generating one batch with all recipients, we now will generate
one batch per 1000 members and distribute the recipients across them.
ref PROD-233
Errors were not handled properly because of a missing rollback and try/catch.
Using a function is easier generally.
Also added ignored contraint checks to increase performance a tiny bit.
They ended up not mattering much, so we can consider to remove them again.
refs ARCH-29
- Added Sentry Profiling to collect more detailed performance data on
the backend.
- This feature is opt-in behind a config. To enable profiling, first
enable tracing with `sentry.tracing.enabled: true`, then set
`sentry.profiling.enabled: true` and `sentry.profiling.sampleRate` to a
decimal number between 0 and 1.
I added missing translations for German
- [x] There's a clear use-case for this code change, explained below
- [x] Commit message has a short title & references relevant issues
- [x] The build will pass (run `yarn test:all` and `yarn lint`)
We appreciate your contribution!
closes https://github.com/TryGhost/Product/issues/4230
- deleting a post could cause React components to trigger save tasks during teardown which then threw errors because they attempt to set properties on a deleted model instance
- added checks to the `beforeSaveTask()` to abort if the post object has been deleted
closes https://github.com/TryGhost/Product/issues/4228
- when leaving the editor via back button the feature image caption editor's blur handler was called by the React editor component after Ember had torn down the route resulting in an attempt to use `post.set()` when `post` doesn't exist
- the error also caused Lexical to re-render to try and recover which then triggered the blur handler again resulting in an infinite loop
- adding a check to ensure `this.post` exists was enough to fix the problem
fixes https://github.com/TryGhost/DevOps/issues/123
- ember-auto-import supports webpack v5 and v5 supports persistent file
caching
- this stores a cache of built files across warm boots, so Admin built
via `yarn dev` should be faster next time
- also ensures that recursive `node_modules` folders are emptied upon
`yarn fix`, as the webpack cache is stored in `node_modules` under
`ghost/admin`
- locally, this speeds up a warm build by 2x (!)
refs.
7b40393d77
We're improving the usability and possibilities for publishers to
migrate from other platforms such as Substack, Medium or Mailchimp. This
PR applies changes to Ghost Settings to support the new flows, more
specifically:
- moves import and export functions out of Labs to its own setting,
directly available from search and the menu
- adds direct access to various platform migrations
- moves "Delete all content" to a dedicated setting group at the bottom
of all setting
---------
Co-authored-by: Jono Mingard <reason.koan@gmail.com>
refs TryGhost/Product#4125
This PR adds two new integration tests to ensure all our Koenig cards
are rendered properly after going through the EmailRenderer. Although we
have thorough tests for the cards themselves in the Koenig repo, the
EmailRenderer does post-processing on the rendered HTML, such as
inlining CSS, which can adversely impact the rendered output of our
cards in email clients (usually Outlook).
Since email newsletters are a core feature of Ghost, these bugs are
typically fairly urgent, and since it is email, they are also quite
difficult to troubleshoot and fix. These two tests are intended to
prevent bugs of this sort, which in the past have been created by
seemingly harmless changes like bumping dependencies that are used in
the EmailRenderer.
The idea is to create a 'Golden Post' which has at least 1 of every card
from Koenig, run that post through the EmailRenderer, and take a
snapshot of the rendered HTML. In the future, if we make any changes to
the EmailRenderer or the Koenig cards themselves, this will trigger us
to carefully consider the changes, and it provides an 'expected' output
to compare our changes against.
Additionally, the second test simply checks that all cards from
`kg-default-nodes` are included in the 'Golden Post'. This protects
against any new cards that we will add in the future — as soon as we add
them to Koenig and bump `kg-default-nodes` in Ghost, this test will
fail, prompting us to add the new card to the Golden Post and update the
snapshots.
We should also run the 'Golden Post' through a test in Litmus, which
allows us to visually inspect the rendered email across many different
email clients. Ideally we would create a process to review the output of
the 'Golden Post' in Litmus whenever we update the snapshot as well.
I've just pushed a commit with the Thai translation. This should be a
sufficient translation but there is always room for improvements and a
second pair of eyes.
---------
Co-authored-by: Ryan Feigenbaum <48868107+royalfig@users.noreply.github.com>
I added missing Afrikaans Translations
- [x] There's a clear use-case for this code change, explained below
- [x] Commit message has a short title & references relevant issues
- [x] The build will pass (run `yarn test:all` and `yarn lint`)
We appreciate your contribution!
refs ARCH-25
- Added a log message if the email_count on an email differs from the
totalCount calculated while creating batches by more than 1%, so we can
investigate further.
refs PROD-60, PROD-222, PROD-223, PROD-89, PROD-94
- Indicator shows up in the monthly/yearly toggle if there are any paid
tiers with yearly discount, and shows the highest discount in order to
nudge visitors towards checking out yearly plan
- A couple of smaller portal improvements: typos, spacing, alignment
---------
Co-authored-by: Simon Backx <simon@ghost.org>
refs https://github.com/TryGhost/Product/issues/2795
---
<!-- Leave the line below if you'd like GitHub Copilot to generate a
summary from your commit -->
<!--
copilot:summary
-->
### <samp>🤖 Generated by Copilot at 2a6b633</samp>
This pull request improves the localization of the portal feature for
Serbian users by adding or updating translations in the `portal.json`
file.
closes https://github.com/TryGhost/Product/issues/4227
- added `@onTKCountChange` to `<KoenigLexicalEditorInput>`
- when present the `isTKEnabled` flag will be turned on and the `<TKCountPlugin>` registered
- added `@registerAPI` support to `<KoenigLexicalEditorInput>` so we can focus the caption editor when its TK indicator is clicked
- added manual display of TK indicator for the caption input
- default editor indicator positioning doesn't work for this input because its container is not full editor width
- hid it by adding `overflow: hidden` to the inner caption container
- added custom indicator button shown when we have a non-zero count
no issue
- recently we introduced code that broke the editor in older versions of
Safari but we weren't alerted to it until we started getting customer
reports
- we have an `ErrorBoundary` around the React editor components but this
wasn't reporting the error anywhere and simply showed an error message
- updated the boundary to report to Sentry when configured so we can
notice and fix any editor-breaking issues faster
refs https://github.com/TryGhost/DevOps/issues/119
- members_newsletters needs members_subscribe_events, but it was also
then generating `subscriptions` records and the whole thing was really
slow
- for now, subscriptions is not a used table so we can remove use of it
- also adds support for generating more than one subscription record
with an 80% chance of being subscribed
refs https://github.com/TryGhost/DevOps/issues/119
- this function can simply call the `import` function, which performs
the same code as we had here
- this makes the code cleaner to read and understand
refs https://github.com/TryGhost/DevOps/issues/119
- this switches away from using a static list of names in favor of ones
generated by faker, so we don't run into duplicate names
- also minor code re-arranging
refs ARCH-21
- We currently have NewRelic setup for a few of our largest customers
for monitoring performance, but it is too expensive to enable across all
sites
- Sentry has similar (but simpler) performance monitoring tools to keep
track of response times that are available to us for free, but we just
haven't configured them
- This PR sets up Sentry Performance monitoring for API requests so we
can have one place for monitoring errors + performance so we can stay on
top of response times more easily.
- Tracing is disabled by default, so there is no additional overhead
unless `sentry.tracing.enabled` is set to `true` in the site's config.
Additionally, `sentry.tracing.sampleRate` should be set to a decimal
value between 0 and 1. This value defaults to 0 to avoid accidentally
blowing through quota, and requires a value to explicitly be set in
order to send the traces to Sentry.
refs https://github.com/TryGhost/Product/issues/4209
- bumped Koenig packages
- `koenig-lexical` added nested editor TK support
- all packages dropped Node 16 support
- switched to using `isTKEnabled` prop and `<TKCountPlugin>`
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
fixes PROD-61
This adds a new default plan setting. It defaults to yearly, which is
the current default selected interval in Portal.
Behind the new portal improvements feature flag, the default plan can be
changed. It will also change automatically if the available intervals
are changed.
This PR also wires up passing the new setting to the Portal preview.
Refs https://github.com/TryGhost/Product/issues/4218
- Moving the TK confirmation step into a small modal as a more
lightweight solution
---------
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
refs https://ghost.slack.com/archives/CTH5NDJMS/p1701688836406919
Deleting members with email disabled, results in deleting all members
due to a broken NQL filter.
The filter `(email_disabled:1)` results in selecting all members because
of the surrounding brackets, which cause a `yg` filter to be generated
by NQL which is not supported by code that handles the Mongo filters.
This is a quick fix to reduce damage, this will need a proper fix in NQL
/ lower level.
Added files for swiss german (de_CH). Went for the formal "Sie" (as
opposed to the informal "du" currently used in the locale de).
Suggestion: Allow variants like de_CH@informal and de_CH@formal to make
it possible to be able to offer both, formal and informal addresses for
languages like f.e. german or french and their respective variants.
---------
Co-authored-by: Ryan Feigenbaum <48868107+royalfig@users.noreply.github.com>
fixes https://github.com/TryGhost/Product/issues/4216
When generating page/post metadata, we generate a "context object" from
several meta helpers. In the event there is no context from the resource
type, we generate a fallback object.
However... we generate this fallback object no matter what.
Now, the fallback object is just 3x settingsCache.get, which should be
fast, but it's not. I've opened a separate issue for that: https://github.com/TryGhost/Product/issues/4217
In the mean time, we can switch this logic around to only do these calls
when we have no other context.
From testing, this allows for 10% more throughput on a post 🤯
fixes GRO-72
- added "default_email_address" and "support_email_address" to the
public settings
- when available, use these addresses in Portal. Otherwise, fallback to
current logic
refs https://github.com/TryGhost/Product/issues/4196
The offers API basically returns the data you pass to it, rather than
the created database record. It looks like this is how it was intended
to work in the first place; the `setMilliseconds` is because the test
helper expects `.000Z`, which I assume is because MySQL will strip off
the milliseconds when it's saved.
fixes GRO-71
- Current flow: unchanged
- New managed flow: verification required
- New managed flow with custom sending domain: only verification
required for different domains
- Self hosters (feature flag): no verification required
closes https://github.com/TryGhost/Product/issues/4191
Without this patch, themes can read arbitrary files from your system and
expose them to the internet via the layout feature of express-hbs.
For example `{{!< ../../../../config.production.json}}` would spit out config,
which can contain secrets.
As theme upload is restricted to users with the Admin role, this mostly effects
hosting providers which use their own secret keys for e.g. mail or database config
refs https://github.com/TryGhost/Product/issues/4181
We were seeing slow queries when joining on this table, and the index
speeds them up. The down migration is tricky because when we add the
index MySQL can optimise away some `KEY` indexes on the `newsletter_id`
column. When we then go to remove the newly created index, there is no
index for the FK!
We also remove the use of `force index` as 1. the index we're forcing is
optimised away and 2. we don't need it anymore!
Co-authored-by: Daniel Lockyer <hi@daniellockyer.com>
refs GRO-80
- added a new meta field "email_verified" to the /verification endpoint
for newsletters. This meta field contains which email has been verified,
"sender_email" or "sender_reply_to"
- updated copy in newsletter settings, based on which email has been
verified
no issue
- bumped `@tryghost/koenig-lexical` to version that no longer uses negative lookbehind in a regex which wasn't supported in Safari until version 16.4
no issue
- bumped `@tryghost/koenig-lexical` to version that no longer uses negative lookbehind in a regex which wasn't supported in Safari until version 16.4
fixes GRO-73
We need to avoid duplicating the complex logic for determining the
default email address and the support email address. So these are now
exposed as calculated settings.
ref GRO-54
fixes GRO-63
fixes GRO-62
fixes GRO-69
When the config `hostSettings:managedEmail:enabled` is enabled, or the
new flag (`newEmailAddresses`) is enabled for self-hosters, we'll start
to check the from addresses of all outgoing emails more strictly.
- Current flow: nothing changes if the managedEmail config is not set or
the `newEmailAddresses` feature flag is not set
- When managedEmail is enabled: never allow to send an email from any
chosen email. We always use `mail.from` for all outgoing emails. Custom
addresses should be set as replyTo instead. Changing the newsletter
sender_email is not allowed anymore (and ignored if it is set).
- When managedEmail is enabled with a custom sending domain: if a from
address doesn't match the sending domain, we'll default to mail.from and
use the original as a replyTo if appropriate and only when no other
replyTo was set. A newsletter sender email addresss can only be set to
an email address on this domain.
- When `newEmailAddresses` is enabled: self hosters are free to set all
email addresses to whatever they want, without verification. In addition
to that, we stop making up our own email addresses and send from
`mail.from` by default instead of generating a `noreply`+ `@` +
`sitedomain.com` address
A more in depth example of all cases can be seen in
`ghost/core/test/integration/services/email-addresses.test.js`
Includes lots of new E2E tests for most new situations. Apart from that,
all email snapshots are changed because the from and replyTo addresses
are now included in snapshots (so we can see unexpected changes in the
future).
Dropped test coverage requirement, because tests were failing coverage
locally, but not in CI
Fixed settings test that set the site title to an array - bug tracked in
GRO-68
refs TryGhost/Product#4175
- Added error handling to Sentry's beforeSend function in both Admin and
Core, so if there is any error in beforeSend, we will still send the
unmodified event to Sentry
- This is in response to an incident yesterday wherein the beforeSend
function threw an error due to an unexpected missing value in the
exception. The event sent to Sentry was the error in the beforeSend
function, and the original error never reached Sentry.
- If the original event had reached Sentry, even if unmodified by the
logic in beforeSend, we could have been alerted to the issue sooner and
more easily identified all affected sites.
- Also added defensive logic to protect for certain values in the
exception passed to beforeSend not existing and added unit tests for the
beforeSend function in admin and core
no refs
This reverts commit 329488139a.
- updates for mobile result in the locator queries finding two elements
instead of one, causing tests to fail
- reverting for now until we take a look closer with mobile, as it seems
CI is running at a mobile size
refs https://github.com/TryGhost/Product/issues/4184
- set up property on the editor controller for tracking number of TKs, action for updating it, and reset mechanism to ensure we go back to 0 when switching post
- uses random number for now pending `<TkPlugin>` being updated to expose the TK count
- passed TK count data to the publish flow modal so it can show a reminder step before the publish options step when there are still TKs in the post content
- added `onCountChange` prop to `<TkPlugin>` ready for the count feature to be implemented
refs https://github.com/TryGhost/Product/issues/4181
- we've seen MySQL change its query planner to use a different index
than the ideal one, resulting in drastically slower query performance
when fetching newsletters with the member count
- this forces the use of the ideal index on MySQL
- this kept many of the Ghost team up overnight, so I hope MySQL is
happy
refs https://github.com/TryGhost/Ghost/pull/19057
- the flag and toggle UI had been added but we were missing the final part of wiring up the feature in Admin and passthrough to the editor
fixes GRO-34
fixes GRO-33
This is a revision of a previous commit, that broke the browser tests
because changes in the data generator (requiring bookshelf had side
effects).
This adds a new way to run all tests with enforced numeric ObjectIDs.
These numeric ids cause issues if they are used withing NQL filters. So
they surface tiny bugs in our codebase.
You can run tests using this option via:
NUMERIC_IDS=1 yarn test:e2e
Removed some defensive logic that could be explained by this discovered
issue.
refs https://github.com/TryGhost/Arch/issues/101
Refined the cache invalidation logic so that when updating a user, we
only invalidate the cache when an attribute of the user that is used on
the frontend changes.
refs https://github.com/TryGhost/Product/issues/4152
---
<!-- Leave the line below if you'd like GitHub Copilot to generate a
summary from your commit -->
<!--
copilot:summary
-->
### <samp>🤖[[deprecated]](https://githubnext.com/copilot-for-prs-sunset)
Generated by Copilot at a28462f</samp>
This pull request adds a new admin-x app called `admin-x-demo`, which
demonstrates how to use the shared packages `admin-x-framework` and
`admin-x-design-system` to create a simple app that renders a button and
a modal. It also improves the development workflow, the vite
integration, the dependency management, and the type checking for the
admin-x apps and packages. It modifies some files in the
`admin-x-framework` and `admin-x-design-system` packages to make the
modals prop optional, to introduce a new type for the props from the
Ember app, to fix the z-index of the modal backdrop, and to use
consistent file extensions and module syntax.
fixes GRO-34
fixes GRO-33
This also adds a new way to run all tests with enforced numeric ObjectIDs.
These numeric ids cause issues if they are used withing NQL filters. So they
surface tiny bugs in our codebase.
You can run tests using this option via:
NUMERIC_IDS=1 yarn test:e2e
Also removed some defensive logic that could be explained by unquoted ids.
refs https://github.com/TryGhost/Product/issues/4140
- added `social-image` image size to our `internalImagesSizes` list with a max-width of 1200
- extracted image utils from `{{img_url}}` helper to a utils file for re-use
- updated `getImageDimensions` method that reads image dimensions and modifies the finalised `metaData` object before use to adjust dimensions and associated URLs to match max width of 1200px
refs https://github.com/TryGhost/Product/issues/4159
---
<!-- Leave the line below if you'd like GitHub Copilot to generate a
summary from your commit -->
<!--
copilot:summary
-->
### <samp>🤖[[deprecated]](https://githubnext.com/copilot-for-prs-sunset)
Generated by Copilot at 9e68f4d</samp>
This pull request refactors several components in the `admin-x-settings`
app to use common hooks from the `@tryghost/admin-x-framework` package,
which reduces code duplication and improves consistency. It also updates
the `package.json` file and adds unit tests for the `admin-x-framework`
package, which improves the formatting, testing, and dependency
management. Additionally, it makes some minor changes to the `hooks.ts`,
`FrameworkProvider.tsx`, and `.eslintrc.cjs` files in the
`admin-x-framework` package, which enhance the public API and the
linting configuration.
