refs https://linear.app/tryghost/issue/DEV-23/workaround-for-yarn-caching-issues
- in our build pipeline, we add some more dependencies for our internal
adapters
- recently we've been seeing caching issues with these dependencies, not
sure why
- to workaround that, we'll just include them here and eventually bring
the adapters into the OSS repo
ref https://linear.app/tryghost/issue/ENG-1466
ref https://linear.app/tryghost/issue/ENG-1484
- Previously, filtering members with multiple "Unsubscribed from
newsletter x" led to no filtering at all, all members were returned
- This was caused by a bug in NQL, that is fixed in version 0.12.5, cf.
[commit](dd18d1d6ca)
- We're also removing the safeguard in the product around bulk deletion
when multiple newsletter filters are in use, as the root problem has
been fixed
ref
https://linear.app/tryghost/issue/ENG-1543/debounce-the-members-lastseenatupdater
- The `LastSeenAtUpdater.updateLastSeenAt` function is called in
response to a `MemberClickEvent` — when a member clicks a link in an
email with tracking enabled. This function can be called many times for
the same member in a short period of time if e.g. a link checker is
clicking all the links in an email they received.
- This function should only update a member's `last_seen_at` timestamp
once per day. To accomplish this, `updateLastSeenAt` runs a
`select...for update` query to find the member's current `last_seen_at`
timestamp, and only updates the timestamp if the current `last_seen_at`
is before the start of the current day. The `for update` is required to
avoid a race condition, which previously caused this function to update
the `last_seen_at` timestamp more frequently than needed, which results
in many unnecessary database queries. However, we still run the initial
`select...for update` query for each event, which seems to be resulting
in contention for locks on the member's row in the `members` table.
- This commit introduces a simple in-memory cache so that we avoid
calling `updateLastSeenAt` if the member's `last_seen_at` timestamp has
already been updated in the current day, which should avoid running so
many `select...for update` queries and locking the `members` table up.
ref
https://linear.app/tryghost/issue/ONC-217/implement-the-deliverytime-option-in-mailgun-api-calls
Ghost experiences its highest peak load immediately after sending out a
newsletter, as it recieves an influx of traffic from users clicking on
the links in the email, a burst of email analytics events to process
from mailgun, and an increase in organic traffic to the site's frontend
as well as the admin analytics pages. The `BatchSendingService`
currently sends all the batches to Mailgun as quickly as possible, which
may contribute to higher peak loads.
This commit adds a `deliverytime` parameter to our API calls to Mailgun,
which allows us to specify a time in the future when we want the email
to be delivered. This will allow us to moderate the rate at which emails
are delivered, and in turn that should moderate the peak traffic volume
that Ghost receives in the first 2-3 minutes after sending an email.
The `deliverytime` is calculated based on a configurable parameter:
`bulkEmail.targetDeliveryWindow`, which specifies the maximum allowable
time (in milliseconds) after the email is first sent for Ghost to
instruct Mailgun to deliver the emails. Ghost will attempt to space out
all the batches as evenly as possible throughout the specified window.
For example, if the targetDeliveryWindow is set to `300000` (5 minutes)
and there are 100 batches, Ghost will set the `deliveryTime` for each
batch ~3 seconds apart.
no issue
- These retries were added a while ago when these tests were flaking in
CI, but this only serves to cover up any potential problems with these
tests or the code they are testing
- Ran these tests in CI three times in a row without them failing, so I
don't think the retries are necessary
no issue
- One of the tests in this suite added a member and didn't clean it up
when it was finished.
- Because of this, the tests after this one depended on this test
running first, so running an individual test in isolation might fail,
despite passing when run in the whole test suite
- This commit removes the added member, so all the tests in this suite
should pass whether run independently or all together
ref https://github.com/TryGhost/Ghost/pull/20835
- reimplemented email analytics changes that prioritized opened events
over other events in order to speed up open analytics
- added db persistence to fetch missing job to ensure we re-fetch every
window of events, especially important if we restart following a large
email batch
We learned a few things with the previous trial run of this. Namely,
that event throughput is not as high as we initially saw in the data for
particularly large databases. This set of changes is more conservative,
while a touch more complicated, in ensuring we capture edge cases for
really large newsletter sends (100k+ members).
In general, we want to make sure we're fetching new open events at least
every 5 mins, and often much faster than that, unless it's a quiet
period (suggesting we haven't had a newsletter send or much outstanding
event data).
ref https://linear.app/tryghost/issue/ONC-296
Our `stripe_prices.nickname` field had a length of 50 chars which meant we could error out trying to save a donation Stripe price with a generated product nickname containing a long site title.
- updated db schema and added a migration to change column length to 255
- added truncation to nickname generation to enforce a limit of 250 chars to match Stripe's limit
no issue
Give your audience a simple way to support your work with one-time payments, no membership required.
- cleaned up `tipsAndDonations` labs flag
closes https://linear.app/tryghost/issue/PLG-190
- often when adding portal links to your own site pages the URLs are added as absolute on the site's homepage due to copy+paste from displayed URLs in Admin
- when clicking absolute portal URLs the homepage is first loaded before the Portal popup is shown resulting in a slower and flashier experience
- added a transform for all local portal URLs on the page when Portal is initialized so links open the Portal popup immediately on the current page
ref https://linear.app/tryghost/issue/DEV-20/faster-builds
- this was previously causing duplicate builds of the TS projects
because Nx was building all projects with `build` targets, and we were
also calling `build:ts`
- this cuts 12 compilation jobs from the archive process, which should
help with container build times
closes
https://linear.app/tryghost/issue/ANAL-9/initial-tracker-in-ghost-head
- Given that all of the correct config is in place, output a tracking
script
- This allows us to send pageview events into tinybird
- All of the details (location of the script, destination etc) are kept
in config so that it's easy to change for different environments
ref DES-706
* After a user publishes or schedules a post, they are directed to the post list
* If a post is sent as an email, they are directed to the Analytics page
* In both cases, a confirmation modal is shown
* If a post is published, they can share it directly from the confirmation modal
* Added a "Share" button and some additional functions (view, edit, and delete post) to
published posts in post analytics
* Added a manual "Refresh" button to post analytics so that there is
no need to reload the whole app to update the data
---------
Co-authored-by: Sag <guptazy@gmail.com>
no ref
This service can get rather noisy when doing local development with our
data generator, as we do not use real urls, and therefore generate a lot
of not found errors in the console.
closes https://linear.app/tryghost/issue/ANAL-8/flag-and-config
- This checks if tinybird:stats is set, and if so passes through the
config that is set via the config API
- This is used by Ghost admin to configure where to pull charts from
- we added NestJS to Ghost as a way forwards for a new framework within
Ghost but we haven't added much to it
- requiring all the NestJS code adds about 6-9% to our boot time, so if
we're not using it, it's just time we're burning for no benefit
- for now, I've gated this behind an env var to prevent it from loading
- we can't use labs flags in the boot process, so I've gone for an env
var
ref https://linear.app/tryghost/issue/ONC-274
This fix is done at the endpoint layer, which isn't ideal, but has the smallest
surface area for the change. I think we may want to move it up a layer though,
despite the extra complications. If we move the check _into_ the MembersAPI
class however, we start to run into issues with circular dependncies due to the
mess that is our dependency injection approach with the Members service
ref https://linear.app/tryghost/issue/ONC-274
This test checks that a welcome email is not sent when adding a member via the
Admin API if the site is in need of email verification, regardless of whether
or not the flag to send an email is set.
It is currently failing to demonstrate the whole in our logic.
All tests after this one were in a non-sending email state, which luckily
doesn't affect these tests, but it will affect future ones! This just cleans up
the settings so that they're back to a standard default.
- Swap the e2e config API test to use our newer framework, and match against a
snapshot for the default case
- Move the individual test cases to unit tests (new file) - there are more to add
here, but this is parity with what we had before
- We use unit tests for checking through various cases for how config
changes modify the output as this is faster and more explicit
ref
639be25f1d
ref
f705dda314
- These items are never returned from the API as they've been removed
from the serializer
- The tests also check that they are not present
- They were removed by the referenced commits, which changed how these
things were built
ref PLG-160
- Refactored donation handling logic to be processed within the
`checkout.session.completed` webhook event.
- Added support for capturing and storing donation messages from Stripe
sessions.
- Integrated donation messages into the email notifications sent to
staff.
- Added database integration.
- Removed redundant donation logic from the invoice.payment_succeeded
webhook, since custom fields isn't supported.
- Updated and added new tests
---------
Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
no issue
- when redirecting from Stripe back to Ghost after making a donation the URL contained a double slash (`//#/portal/...`) which triggered browser security errors when Portal modified the browser history stack when navigating
- the above could prevent the donation success modal from closing
ref https://linear.app/tryghost/issue/ENG-1518
After releasing the analytics job improvements, it appears for large
sites we're awfully close to missing some Mailgun events because of an
unexpected behavior of the aggregateStats call for just the opened
events job. This is taking 2-5x(+) the amount of time that the aggregate
queries take for the other jobs, despite not being dependent on the
events.
To err on the side of caution, we're going to roll this back and look to
optimize the aggregation queries before re-implementing. And we may be a
bit more cautious in giving _some_ but not _all_ priority to the
`opened` events.
ref https://linear.app/tryghost/issue/ENG-1518
After releasing the analytics job improvements, it appears for large
sites we're awfully close to missing some Mailgun events because of an
unexpected behavior of the aggregateStats call for just the opened
events job. This is taking 2-5x(+) the amount of time that the aggregate
queries take for the other jobs, despite not being dependent on the
events.
To err on the side of caution, we're going to roll this back and look to
optimize the aggregation queries before re-implementing. And we may be a
bit more cautious in giving _some_ but not _all_ priority to the
`opened` events.
ref
https://linear.app/tryghost/issue/ENG-1505/start-monitoring-event-loop-utilization-in-production-with
- The two main constraints we've observed in Ghost are the database connection pool and the CPU usage. However, there is a third constraint that we may be hitting, but can't currently observe: the event loop.
- This commit re-enabled OpenTelemetry (behind a config flag), removes the problematic tracing instrumentation which was breaking the frontend, and adds a Prometheus endpoint to export the eventLoopUtilization metric.
- This should give us visibility into whether we are hitting constraints in the event loop and address the root cause if we are.
ref KTLO-1
These tokens should prevent untargeted attacks, as the magic link
endpoint needs a token that was generated by the server, similar to a
CSRF token, but without needing any server-side state, or a cookie to
be set for unauthenticated users.
ref https://linear.app/tryghost/issue/ENG-952
- added persistence to the job timestamps
This set of changes reduces the potential for gaps in our email event
processing by adding persistence to the job timestamps. This avoids
expensive queries on the `email_recipients` table after every boot, and
reduces reliance on fallbacks in periods of heavy processing or reboot.
This is our first use of the jobs table to create a persistent line,
instead of its initial use case of single-run jobs. We may expand this
capability and move to use of the jobs model over knex.raw in order to
make this a bit friendlier.
Note: this works with sqlite but datetimes are stored as ints. It still
works fine. https://github.com/knex/knex/pull/5272
ref
https://linear.app/tryghost/issue/ONC-242/frontend-routing-prioritizes-collections-over-taxonomies
- Under a fairly specific edge case with a collection route that conflicts with a default, built-in route ("taxonomy" — like tags, authors, etc), the frontend routing would prioritize the collection over the taxonomy.
- For example, with the following in a custom `routes.yaml`:
```
collections:
/:
permalink: /{primary_tag}/{slug}/
template: index
```
If a post exists with the same slug as its primary tag's slug, the frontend routing would redirect the `/tag/{slug}/` route to the post in the collection, rather than serving the tag itself.
- This commit changes that, so if a collection's route conflicts with e.g. a `/tag/{slug}/` default route, Ghost will still return the built in route, rather than the collection.
ref https://linear.app/tryghost/issue/ENG-1477
- updated email analytics job to prioritize open events
- put limits on non-open event fetching
- updated job to now restart itself until processing is at a
sufficiently low volume
Previously the EmailAnalytics job would process all event data equally.
When there's sufficient recipients (>20k), we could see delays in the
open rate data in Admin because of all the delivered events being
processed. Open events are far more important to users, so we've now
prioritized processing those events before any others.
Processing of events shouldn't be any faster or slower with this as this
doesn't change throughput, just order.
NOTE: Use the mailgun-mock-server in TryGhost/Toolbox for testing.
ref https://github.com/TryGhost/Ghost/pull/20793
- now that there's a non-zero suggested value amount Stripe's UI shows a button to change the amount rather than showing the amount input field immediately
- added extra click to the tests and an expectation that the default value is set correctly
closes https://linear.app/tryghost/issue/PLG-156
- updated all default fixtures to use `500` ($5) as the default suggested donation value
- added migration to update existing settings using the old default of `0` to `500`
- this is fine to apply because the feature hasn't been released so there's no explicit `0` values in the wild
- added an acceptance test for the adminx-settings tips & donations section
ref DES-609
- On top of bookmark card, this update adds default background color and font to file, audio, and product cards
- The main purpose of this is keeping embedded look across these cards consistencly regardless of the theme background color and default font
- Themes can still override the styles
ref [ENG-661](https://linear.app/tryghost/issue/ENG-661/)
ref [ONC-253](https://linear.app/tryghost/issue/ONC-253/)
ref [PLG-174](https://linear.app/tryghost/issue/PLG-174/)
- restored the original but reverted fix for unsaved changes modal from https://github.com/TryGhost/Ghost/pull/20687
- updated code to remove some incorrect early-falsy-return logic in `editorController.hasDirtyAttributes` that prevented save of unsaved changes on the underlying model (e.g. excerpt)
- updated unit tests so they are testing real post model instances and therefore are testing what we expect them to test
- added acceptance tests to ensure autosave is working for title and excerpt fields
---------
Co-authored-by: Ronald Langeveld <hi@ronaldlangeveld.com>
ref [ENG-661](https://linear.app/tryghost/issue/ENG-661/)
ref [ONC-253](https://linear.app/tryghost/issue/ONC-253/)
ref [PLG-174](https://linear.app/tryghost/issue/PLG-174/)
- restored the original but reverted fix for unsaved changes modal from https://github.com/TryGhost/Ghost/pull/20687
- updated code to remove some incorrect early-falsy-return logic in `editorController.hasDirtyAttributes` that prevented save of unsaved changes on the underlying model (e.g. excerpt)
- updated unit tests so they are testing real post model instances and therefore are testing what we expect them to test
- added acceptance tests to ensure autosave is working for title and excerpt fields
---------
Co-authored-by: Ronald Langeveld <hi@ronaldlangeveld.com>
DES-696
We upgraded the AdminX Design System to use Radix UI components. However
browser tests fail for checkboxes at the moment which must be fixed for
release.
ref [ENG-661](https://linear.app/tryghost/issue/ENG-661/) ONC-253
- Reverts the revert of
93cbb94b90
of the intial bug fix.
- Updated hasDirtyAttributes logic to ensure the dirty state changes
when typing a draft, despite not title.
- Updated tests and added tests missing from the hasDirtyAttributes
logic
ref ENG-1490
- Regression from the secondary lexical instance.
- Replaced the visibility and positioning styles with width, height, and
overflow to prevent layout space issues.
- Ensured the element takes up no space and is not visible, while
allowing proper initialisation.
no issue
- added passthrough of `feature.contentVisibility` when rendering Lexical to HTML to allow for labs-feature specific rendering of HTML cards
- updated golden post email render test to account for labs flags being on/off
closes https://linear.app/tryghost/issue/PLG-15
- removed `internalLinking` GA labs flag
- renamed search providers to `flex` and `basic`
- keeps old search provider around as it can handle non-English languages unlike the faster flex provider
- updated `search` service to switch from `flex` to `basic` when the site's locale is not english
- bumped Koenig packages to switch from a feature flag for toggling internal linking features to the presence of the `searchLinks` function in card config
- updated tests to correctly switch between flex and basic providers in respective suites
ref ONC-225
- Wires up the `editor_default_email_recipients` key to the settings
public / content api endpoint.
- This key is then wired up to Portal to determine whether it's hiding or
showing the Member subscribe toggle
ref
https://linear.app/tryghost/issue/ENG-1440/backfill-offer-redemption-data-with-a-migration
There was a bug that caused offer redemptions to not be recorded in the
database for some subscriptions that were created with an offer.
However, we still have the `offer_id` attached to the subscriptions, so
we are able to backfill the missing redemptions. The bug was fixed in
bf895e6e99
This commit only contains a migration, which queries for subscriptions
that have an `offer_id` but do not have any offer redemptions recorded,
and adds any missing redemptions to the `offer_redemptions` table.
no ref
- while reviewing the newsletter flows, it was apparent that we were
missing test coverage
Some of the tests in Portal are a bit redundant with tests added for
child components, but it didn't seem worth removing them after getting
them to work. There was a bug in our Portal fixture data that requires a
few changes, as well as some small adjustments for making tests easier
(testing-lib-react has `getByTestId` and simply a `querySelector` to use
alternate test attributes).
REF MOM-315
- Changed to column layout
- Fixed broken currency dropdown
- Included a link to Stripe terms & conditions
- Renamed from "Tips or donations" to "Tips & donations"
Right now identity tokens can only be fetched by the Owner, which means they
implicitly have the Owner role, but we want to expand that. The first step is
adding the role to the token, and then we need to update each place which uses
the token and add an assertion that the role is correct.
- right now, it loops through all packages serially, which isn't
effectively using multi-core machines
- by using `concurrently`, we can rely on it to use all the cores it
can, so this should dramatically speed up the bundling step
ref DES-571
- padding does not work well with paragraph inside blockquote as
horizontal spacing on Outlook
- using margin instead of padding makes sure the spacing is consistent
across Outlook versions
ref DES-571
- iOS Mail app ignores spacing on the \<blockquote\> element, but will
respect spacing on the \<p\> element inside it
- for that reason, we started to enforce always rendering \<p\> inside
\<blockquote\> for emails
- these changes move the spacing related styles from blockquote to p
inside
closes https://linear.app/tryghost/issue/ENG-1432
- bumps `kg-lexical-html-render` package with required rendering change
- bumps `koenig-lexical` with TypeError fix and improved handling of failed image uploads
- bumps other packages that were missed from previous bumps, fixes split versions of underlying lexical packages
ref e626dd9
There has been some flakiness in Github CI with the new tests for the
probe library. We'll start with extending timeouts in case CI is running
particularly slowly.
ref https://linear.app/tryghost/issue/ENG-1254
- when a subscription is canceled automatically by Stripe (e.g. due to
multiple failed payments), we now send a staff notification
- logic before: if a member cancels a sub in Portal, then send a staff
notification
- logic now: if a subscription was active, but is now set to cancel
immediately or at the end of the billing period, then send a staff
notification.
- with that logic change, we now send a cancellation staff notification
when:
1. A member cancels their sub in Portal (existing)
2. A staff member cancels a member sub in Stripe (new)
3. A staff member cancels a member sub in Admin (new)
4. A sub is canceled automatically by Stripe because of multiple failed
payments (new)
- the copy of the staff notification email has also been updated to take
into account 1) manual vs automatic cancellations, and 2) immediate vs
end of billing period cancellations
ref https://linear.app/tryghost/issue/ENG-1408/
- added additional safeguards to the image size dimensions probing
For some reason that requires further investigation, the
probe-image-size package was silently failing (neither resolving nor
rejecting) for a particular URL. This was causing Ghost to hang on to
serving the request, and after a few of these came in, ultimately caused
Ghost to stop being responsive.
Rather than trying to patch a dependency, we'll wrap the call to this
package and use the same timeout we pass into the package (which is
ignored in this particular case) as an additional safeguard.
ref DES-263
- we've recently started forcing white background color to the bookmark
card by default
- the reason was making it look good regardless of the site background
color
- it caused an issue to some sites, mostly in dark mode, because the
text color was inherited from the theme
- this sets explicit color to the bookmark content which is consistent
with the nft card
refs
[ENG-1355](https://linear.app/tryghost/issue/ENG-1355/site-boot-cycling-due-to-free-tier-having-a-currency)
Fixed data importer allowing invalid free product to be imported which
could cause Ghost to not start due to the error:
`ValidationError: Free Tiers cannot have a currency`
It should not be possible to import a free product with pricing data (as
that means its not free 😄)
ref https://linear.app/tryghost/issue/ENG-1266
- Mexico changed tz to not participate in DST
- our package was a couple years behind, so we likely have fixes for
other countries/regions, too
ref https://linear.app/tryghost/issue/ONC-154
- the query params did not carry through on portal sign up links because
of the hash creating an ignored fragment
(/#/portal/signup?ref=something)
Now when we check link attribution, we'll attempt to run the same logic
for the referrer source after stripping out `#/portal` from the URL.
Otherwise we should continue to treat these fragments as fragments to be
ignored by the client.
NOTE: We do not have e2e tests that cover member signup on the front end
and the data entered in the back end. The tests we have mock only the
server side of things. The test added here only covers the data that is
generated from the front end request (at this time), *not* the front end
request itself, meaning it's fragile.
ref https://ghost.slack.com/archives/CTH5NDJMS/p1720422460943619
- bumping 'lexical' from 0.13.1 to 0.14.2 created a few selection bugs
in the editor
- this commit reverts 'lexical' back to 0.13.1 and any related changes
in the editor codebase
ref ENG-945
- Fixed an issue where upload a broken redirects yaml will override the
last working yaml.
- Instead it will now do the validation before saving and overriding the
yaml.
no issue
- we're no longer making use of the websockets experiment so it's just bloat
- this is the whole feature in a single commit in case we need to revive it at some point
- this change contains the removal of the `promise.allsettled` package,
as this is not needed on Node 12+, which removes 75 further dependencies
in production mode
ref https://linear.app/tryghost/issue/ENG-1254
- we currently only store a cancellation reason when a member cancels
manually in Portal
- we now also store "Payment failed" when the cancellation is automatic
due to several payment failures
refs
[ENG-709](https://linear.app/tryghost/issue/ENG-709/%F0%9F%90%9B-bad-redirects-causing-container-tear-down)
Added validation to prevent RegEx's susceptible to ReDoS from being used
with custom redirects. Also moved error details out of `context` and
into `errorDetails` to be consistent with error logging elsewhere as
well as fix issue in admin-x where blank screen would be shown when an
error occurred during redirects upload (due to logic not accounting for
`context` being an object)
closes https://linear.app/tryghost/issue/ENG-1255
- updated Koenig packages including:
- addition of `/preview` for public preview card
- fix for HTML import from Google Docs
- fix for embed thumbnails being cut off in email
- fix for wide image card width on medium screens
- multiple fixes for unhandled (but non user-visible) errors causing noise in console and error logging
ref https://linear.app/tryghost/issue/SLO-173/removed-distinct-from-member-count-query
Performance of GET /members API can be improved by dropping the distinct from the total members count query.
select count(distinct members.id) as aggregate from `members`; // 275ms
select count(*) as aggregate from `members`; // 30ms
In this case we know that the result set will always be unique.
ref https://linear.app/tryghost/issue/ONC-111
- added composite index to posts_tags for post_id,tag_id for faster
lookup
- added composite index to posts for updated_at; this is commonly used
by get helpers on the front end to display data like the latest posts
In testing, this provided a very dramatic improvement for simple get
helper requests like 'filter="id:-{{post.id}}+tag:sampleTag" limit="3"'
which are by default sorted by updated_at desc. I'm not entirely clear
why when sorting by published_at we do not need a composite index - so
far it doesn't seem to be necessary. This should cover the primary cases
for get helpers - the latest posts with a given tag or set of tags.
ref https://linear.app/tryghost/issue/CFR-43/
ref 9d9a421
We recently stopped `select *` from posts when making Content API
requests. This is now being applied to the pages endpoint to help
improve performance. These fields were already being stripped out in the
output serializer, and they will now no longer be returned from the db
at all, reducing the amount of data transferred.
ref https://linear.app/tryghost/issue/CFR-35
- performance improvement intended for the content api/get helpers
The posts table is shared by posts and pages and seldom is queried for
both. It makes sense to add an index on type, and from the perspective
of the content API, also on status as you're almost only ever querying
for published posts or published pages.
closes https://linear.app/tryghost/issue/ONC-115
- OpenTelemetry was throwing errors when viewing posts
- disabled the instrumentation in development mode so it requires explicit config to enable
