We've wrapped both changes in a try/catch to make sure this has no
adverse affects. The endpoint currently doesn't exist - we're only
adding this to get an idea of how much traffic we'll expect to see.
Long term we'll want to read the endpoint from the webmention service.
closes https://github.com/TryGhost/Team/issues/2276
Portal had died with an unintelligible error about portal plans/includes being undefined when there was missing site data in some extreme edge cases. This change catches any errors in site transformation and logs it to console instead of crashing portal unexpectedly
- in the event the Mailgun config doesn't exist, we return `null` from
this function
- this updates the jsdoc to correct the return type of `getInstance`
no refs.
Post feedback popups were not optimised for mobile usage at all. All the content was on top of the window which is really hard to reach using a single hand/thumb and it just looked like a scaled down version of a desktop modal.
closes https://github.com/TryGhost/Team/issues/2206
- removes `www.` from the url shown on links table in post analytics
- we had previously removed http(s) protocol from it as well, and they are only shown while editing the url
This introduces the new suppressions feature which will automatically
unsubscribe members from newsletters when their email is added to the
suppression list in Mailgun, this is usually due to emails either
permanently bouncing to the address, or the member making a spam
complaint.
Both Members and Admins are able to see that the email has been added to
the list, and Members are be able to request their email be removed from
the list via Portal.
Overall this feature should improve delivery rates of newsletters and
improve the rating of the domain you're sending from.
closes https://github.com/TryGhost/Team/issues/2338
If a site has the Free tier hidden from the Portal, and subsequently the Stripe connection is disconnected, this produces a dead-end state where no new members can sign up and the Free tier cannot be reactivated again in Portal settings as its hidden. This change -
- enables free tier toggle to be always shown on site irrespective of Stripe connection
fixes https://github.com/TryGhost/Team/issues/2383
A user could use `{uuid}` inside an email only content and it would work. This currently isn't supposed to be used outside internal features (link click tracking, feedback buttons). For now this is only fixed in the new email flow under the email stability flag.
fixes https://github.com/TryGhost/Team/issues/2339
The email service is now fully covered by tests, and this commit also forces the test coverage to remain 100% after future changes.
closes https://github.com/TryGhost/Team/issues/2011
- Gives publishers the ability to filter members based on which offer they used (redeemed) when they subscribed for a paid membership.
- On the offers page, the redemption count number links to a the members page with the filter already applied making it easy to have insight on which members used the offer / coupon.
We have been adding emails to the suppressions table which are not on
the suppression list in Mailgun due to a misunderstanding of how
Mailgun handles 5xx error codes.
We're seeing behaviour from Mailgun where permanent failures with a
5xx error code are not being added to their internal suppression list,
which is resulting in the Ghost list becoming out of sync with
Mailgun.
Rather than adding emails to the suppression list when Mailgun does,
we're instead going to add emails _after_ Mailgun does, by waiting for
an error code which tells us the email is already on the suppression
list.
Those codes are 605 for previous bounces and 607 for previous spam complaints.
We have been adding emails to the suppressions table which are not on
the suppression list in Mailgun due to a misunderstanding of how
Mailgun handles 5xx error codes.
We're seeing behaviour from Mailgun where permanent failures with a
5xx error code are not being added to their internal suppression list,
which is resulting in the Ghost list becoming out of sync with
Mailgun.
Rather than adding emails to the suppression list when Mailgun does,
we're instead going to add emails _after_ Mailgun does, by waiting for
an error code which tells us the email is already on the suppression
list.
Those codes are 605 for previous bounces and 607 for previous spam complaints.
fixes https://github.com/TryGhost/Team/issues/2398
There was an error when fetching the existing email recipient failure. It ended up matching all recipient failures. The result was that only one failure was stored in the database.
refs https://github.com/TryGhost/Team/issues/2371
- cleans up and adds comments for portal playwright tests
- updates data test attributes for portal trigger and popup selectors for consistency
- updates data attribute usage for offers
refs https://github.com/TryGhost/Team/issues/2371
- in case all tiers are archived before new tier is created, the add tier section can be collapsed and will need to be opened first before going through add tier flow
refs https://github.com/TryGhost/Team/issues/2393
- During boot and loading the active theme, we now cache the result of
the gscan validation. Cache configuration can happen in
`adapters.cache.gscan`
- We now also return non-fatal errors when activating or adding a theme.
- When the `themeErrorsNotification` feature flag is on, we fetch the
active theme (which includes the validation information) when loading
admin
- If the currently active theme has errors, we show an error
notification that can open the error modal
- Added a new endpoint: `/ghost/api/admin/themes/active/` that returns
the result of the last gscan validation of the active theme. If no cache
is available, it will run a new gscan validation.
- Added new permissions for the active action/endpoint (author, editor,
administrator)
closes https://github.com/TryGhost/Toolbox/issues/497
- The classification of fatal/non-fatal errors has been updated to only be fatal when causing page renders with 5xx or 4xx responses.
- Some of the rules checking Ghost 5.x compatibility have been relaxed to only be "error" with the gscan version bump
- You can find more details on which exact rules were relaxed in the gscan's commit log - https://github.com/TryGhost/gscan/compare/v4.35.1...v4.36.0
refs https://github.com/TryGhost/Toolbox/issues/497
- During gscan fatal error downgrade to non-fatal some of the deprecated helpers were a bit vague to debug with no information on which exact "resource" was invalid
- Added resource name to the log for clarity. Should make life easier when debugging potential get helper misuses
refs https://github.com/TryGhost/Toolbox/issues/406
refs b2a3e03ef3
- The "code" property in the global "error" variable (accessible via {{error.*}}) has been long deprecated - time to go, bye!
- When {{code}} or {{error.code}} helpers are used in the templates they will output an empty string from now on. Use {{statusCode}} instead!
- the test was using incorrect test state that was copied over from adding label test
- also adds guard for empty newsletters in member filters as in some cases it might not exist as found by test
fixes https://github.com/TryGhost/Ghost/issues/16057
Briefly, Ghost created two Customer objects via the Stripe API when an
existing subscriber would upgrade to a paid subscription, one in an API
call to create the Customer and then a second as a side effect of an API
call to create a Checkout session for the user. The fix is passing the
reference to the Customer object to the API call to create the Checkout
session; Stripe will no longer redundantly create a Customer object in
this case.
This largely impacts the owner's experience of the Stripe Dashboard; it
will correct their new Customer count (going forward) and make searches
for users by name or email address return one responsive object which
has the actual subscription in it versus returning two and forcing them
to look in each to e.g. refund a transaction or similar.
no issue
When using admin as a user with author or editor permissions, admin tried to load the member counts in order to display them in the menu. But authors and editors are not allowed to see the members. So the request returned a 403.
It is not necessary to load the member counts for authors and editors, so we can just skip the request.
fixes https://github.com/TryGhost/Team/issues/2246
This solution adds some retries when fetching the recipients for a
batch. For an unknown reason the recipients can be empty (while they
aren't in the database). This should fix the issue for now until we find
more information about the root cause.
refs. https://github.com/TryGhost/Team/issues/2393
- a labs flag had to be created so we avoid working in branches
- permanent notification toast was added to make theme errors more discoverable
- static modal was needed to hold theme error details
When Mailgun fails to deliver an email to an address because the
address has already bounced before, it gives us a permanent fail event
with a 605 error code rather than a 5xx one. Because we want to
"backfill" our suppressions data with previously bounced email
addresses, we want to handle this specific error code.
We may update this logic in the future based on new information from
Mailgun with respect to their 6xx error codes and the
meanings/underlying cause of theme.
This also moves the tests which check for whether or not emails are
suppressed into their own fail so that we do not pollute the event
storage tests, and adds more tests cases.
We also fix a leaky sinon stub which we were not resetting in the email
event storage tests
The email_recipient fixtures were using duplicate and mismatched email addresses
rather than having them correctly map to the Members, which is required for testing
email suppressions.
no refs.
- Stripe Connect footer in the Portal settings sidebar was covering settings on small screens
- the border for Stripe Connect box was not visible in dark mode
- the "Save and Close" button background was not visible in dark mode in Portal settings
no issue
With the increased usage of DomainEvents, it gets harder to build
reliable tests without having to resort to timeouts. This utility method
allows us to wait for all events to be processed before continuing with
the test.
This change should speed up tests and make them more reliable.
It only adds extra code when running tests and shouldn't impact
production.
closes https://github.com/TryGhost/Team/issues/2361
If a free trial tier existed on site and its set to 'Invite only' in membership settings, the free trial copy still showed on portal.
- removes free trial copy from portal if site is invite only
- adds playwright test to make sure free trial copy is not shown for invite only sites
- for some reason, Node 18 detects a lower coverage than the configured
threshold so this fails
- I've temporarily lowered the threshold until we can investigate why
refs https://github.com/TryGhost/Toolbox/issues/488
- Node 18 is now LTS so we're adding support for it
- this adds Node 18.12.1 (the latest security release) to our supported
ranges and CI
There are currently two issues with the suppressions table:
- We have some incorrect rows
- We have missing UNIQUE constraints
We want to completely wipe the tables and start fresh, as well as make
sure that the UNIQUE constraints are added, so we drop the table
completely, and then re-add it, which should result in an empty
suppressions table with all expected constraints.
We've also renamed the `email_address` column to `email` to match our
`users` & `members` tables
fixes https://github.com/TryGhost/Team/issues/2366
refs https://ghost.slack.com/archives/C02G9E68C/p1670232405014209
Probem described in issue.
In the old MEGA flow:
- The `email_verification_required` check is now repeated inside the job
In the new email service flow:
- The `email_verification_required` is now checked (didn't happen
before)
- When generating the email batch recipients, we only include members
that were created before the email was created. That way it is
impossible to avoid limit checks by inserting new members between
creating an email and sending an email.
- We don't need to repeat the check inside the job because of the above
changes
Improved handling of large imports:
- When checking `email_verification_required`, we now also check if the
import threshold is reached (a new method is introduced in
vertificationTrigger specifically for this usage). If it is, we start
the verification progress. This is required for long running imports
that only check the verification threshold at the very end.
- This change increases the concurrency of fastq to 3 (refs
https://ghost.slack.com/archives/C02G9E68C/p1670232405014209). So when
running a long import, it is now possible to send emails without having
to wait for the import. Above change makes sure it is not possible to
get around the verification limits.
Refactoring:
- Removed the need to use `updateVerificationTrigger` by making
thresholds getters instead of fixed variables.
- Improved awaiting of members import job in regression test
The MailgunEmailSuppression list was incorrectly adding emails
to the suppression list for permanent failure events which have
an error code outside of the 5xx range.
fixes https://github.com/TryGhost/Team/issues/1996
**Issue**
Our Magic links are valid for 24 hours. After first usage, the token
lives for a further 10 minutes, so that in the case of email servers or
clients that "visit" links, the token can still be used.
The implementation of the 10 minute window uses setTimeout, meaning if
the process is interrupted, the 10 minute window is ignored completely,
and the token will continue to live for the remainder of it's 24 hour
validity period. To prevent that, the tokens are cleared on boot at the
moment.
**Solution**
To remove the boot clearing logic, we need to make sure the tokens are
only valid for 10 minutes after first use even during restarts.
This commit adds 3 new fields to the SingleUseToken model:
- updated_at: for storing the last time the token was changed/used). Not
really used atm.
- first_used_at: for storing the first time the token was used
- used_count: for storing the number of times the token has been used
Using these fields:
- A token can only be used 3 times
- A token is only valid for 10 minutes after first use, even if the
server restarts in between
- A token is only valid for 24 hours after creation (not changed)
We now also delete expired tokens in a separate job instead of on boot /
in a timeout.
refs: 5f90baf6fe
- Ghost has a character limit on post slugs of 191 characters,
- Sometimes, the slug that is generated from the title in Revue content is longer than this, causing the import to fail.
- This PR trims that generated post slug to 190 characters.
closes https://github.com/TryGhost/Team/issues/2380
- improved offer validation for `amount` field to cover all type/amount cases
- added validate-on-blur to the amount field to match our standard validation behaviour
- added re-validation of the amount field when the type is changed and the amount gets reset
- removed the internal parsing of a decimal trial days entry to an integer so the field value matches what is set internally and we let the user know that partial trial days are not supported
Non-user-facing refactors:
- renamed `_saveOfferProperty` to `_updateOfferProperty` to better reflect what it does
- fixed missing indentation for conditional blocks in the offer template
- this was all getting terribly behind so I've done several things:
- majority of `@tryghost/*` except Lexical packages
- gscan + knex-migrator to remove old `@tryghost/errors` usage
- bumped lockfile
refs:
5f90baf6fe
- The check for hasIssuesCSV didn't normalize the filename first,
meaning the importer is super sensitive to zip structure
- This allows for zips that contain a directory, so that it will still
be processed as a revue import, not a Ghost import
refs https://github.com/TryGhost/Team/issues/2235
We found some cases which can cause a site to have member emails that have invalid characters like `member@example.com�`. This happened due to the `validator` version used by Ghost not able to catch some specific cases as invalid email, allowing members to be created with them either via Admin or Importer or direct signup. Portal UI already blocked these email as invalid. This change:
- updates `@tryghost/validator` to include a latest version of email validator that catches these invalid cases
- doesn't allow member creation with invalid email like above
- doesn't allow existing member emails to be edited to invalid
fixes https://github.com/TryGhost/Team/issues/2374
When clicking 'Show active offers' in the archived offers view, it would
always return to an empty offers list, even when there are active
offers.
refs:
5f90baf6fe
- The OG implementation of importing revue subscribers was very naive
- This sures it up to use our proper member importer, which makes sure
everything works perfectly:
- adds an import label
- ensures members are subscribed to newsletters
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
refs: 5f90baf6fe
- Blockquotes without any wrapping tag get converted to Mobiledoc weirdly. Wrapping them in `<p>` tags helps solve that
- Also removes empty paragraph tags which cause unwanted large blank spaces in content
- Remove internal #revue from Revue content
refs: https://www.getrevue.co/app/offboard
- Revue is stopping all paid subscriptions on 20th Dec, and shutting down on Jan 18th.
- This update allows Ghost to accept and handle the zip file Revue are providing as an export in Labs > Importer
- It will import posts (as best as we can with the data provided) and subscribers as free members
- At present it doesn't import paid subscribers, as we don't have that info, but you can disconnect Revue from your Stripe account to prevent all your subscriptions being cancelled & there's the option this can be fixed later
- There will be further updates to polish up this tooling - this is just a first pass to try to get something in people's hands
Co-authored-by: Paul Davis <PaulAdamDavis@users.noreply.github.com>
fixes https://github.com/TryGhost/Team/issues/2386
**Issue:**
- When trying to import a member that already exists, and has
'subscribed' set to 'true' in the CSV, the newsletters the member is
subscribed to are reset to the default newsletters.
- When ediging a member with the API and setting `subscribed` to true,
the same happens.
**Cause:**
A faulty check for the `status` property of a newsletter.
Fixed and added a new E2E test.
- Now that the importer runs in a job, it seems sensble that we should
do this
- If posts are imported with HTML set, but not mobiledoc, we now convert html -> mobiledoc
- Note: This also converts the mobiledoc -> html so _may_ be lossy
- Without this, imports that only have HTML, not mobiledoc, would have
resulted in empty posts, so lossy > empty
no refs
- this adds a theme documentation link to the design settings footer if there's an URL set in the package.json of the theme. An example of the package.json property: "docs": "https://example.com"
closes https://github.com/TryGhost/Team/issues/2012
- Members can now be filtered based on the newsletters they are
subscribed to.
- Defaults to the existing newsletter filtering if user does not have
more than 1 newsletter.
refs: 8ed5f9784d
- When importing content from a JSON file in Settings > Labs, a public tag
like `Import 2022-12-03 19:57` gets added to each newly imported post.
- This tag should not be public. It definitely serves a useful
purpose but has no useful function for readers of the site and should
not be shown to readers.
refs https://ghost.slack.com/archives/C02G9E68C/p1670960248186789
This reverts a change that was made here:
f4fdb4fa6c (r93071549),
but it still moved the original code to a new location in the
LastSeenAtUpdater
It includes a new E2E test to make sure timezones are supported
correctly.
- By not using Bookshelf, we no longer fire webhook calls
- By not using the member repository, we don't fetch and update the
member model and the labels relation in a forUpdate transaction, which
caused deadlock issues on the labels/members_labels tables which were
hard to resolve. Until now I was unable to find the other conflicting
transaction that caused this deadlock. Moving to raw knex (instead of
Bookshelf) and only updating the last_updated_at column should remove
the deadlock issue.
This removed the test for the email service wrapper, since it started
failing for an unknown reason and the test didn't make much sense (was
added earlier only to bump test threshold).
refs TryGhost/Team#2294
- If user enter the slug name and then leave the post page, we should
wait until the slug would be saved on backend. The problem can be
reproduced with slow internet connection.
- The get helper can sometimes take a long time, and in themes that have many get helpers, the request can take far too long to respond
- This adds a timeout to the get helper, so that the page render doesn't block forever
- This won't abort the request to the DB, but instead just means the page will render sooner, and without the get block
refs https://github.com/TryGhost/Team/issues/2371
- test publishes a post with access for a single tier then checks the front-end with no member, member on wrong tier, and member on right tier
refs: https://github.com/TryGhost/Ghost/issues/15725
- our users are having difficulties getting onboarded with mailgun
- we're adding an explicit and unique tag to all requests, to help mailgun detect when mail is being sent from Ghost
closes sentry Admin-423
- there may be times when the mousemove event handler fires when the document is not in a ready state resulting in an attempt to get a document position that doesn't exist
- should fix `Could not find parent section from element node` errors
refs https://github.com/TryGhost/Ghost/issues/14101
- migrated component to Glimmer
- swapped usage of `ValidationState` mixin for `{{validation-status}}` modifier
- updated modifier to accept custom error/success classes
- removed unnecessary/unused code in the `gh-benefit-item.js` backing class
- adds test that cover creating and signing up to multiple-month/forever offers
- checks that the offer information is shown to members during signup and in account detail
closes https://github.com/TryGhost/Team/issues/2382
The preview text is getting set to subject line in the new email flow so it repeats multiple times in the inbox(subject+preview+title). This was because the new flow doesn't use the post serialisation that the old system did, causing excerpt to be empty in the email rendering.
Old system was using post serialisation here -
a721e4f2d7/ghost/core/core/server/services/mega/post-email-serializer.js (L136-L139).
This change adds explicit method to calculate the preview text for email in email renderer service using same logic as used in old system.
Co-authored-by: Simon Backx <git@simonbackx.com>
closes https://github.com/TryGhost/Team/issues/2376#event-8026429598
- if an offer is expired/in past, we no longer show it in member account info against the price
- one-time offers are never showed in portal in member account detail, as the payment information shown to member in Portal points to charge at next payment
- if trial days are over for a subscription, portal doesn't show any offer data on member account detail
no refs
- added two new themes, Taste and Episode to the admin
- updated theme screenshots that weren't up-to-date with the latest changes added to the themes
- some theme screenshot files were JPGs and replaced them with PNGs for consistency
refs https://github.com/TryGhost/Team/issues/2370
Due to a possible bug in either `@sentry/node` (mainly the Express
middlewares and the usage of deprectated Domain) and Node v16+,
unhandled promise rejections are transformed into uncaught exceptions
and cause Ghost to crash in unexpected situations.
Reverting to `v7.11.1` fixes this (but definitely not ideal at all)
because errors are caught in the Express middleware.
Reproduction repo:
https://github.com/SimonBackx/sentry-node-unhandled-rejection-crash
fixes https://github.com/TryGhost/Team/issues/2377
When there is an error thrown that is not a Ghost error, there is no
status code in the error. Calling res.writeHead with an undefined status
code, throws an error and crashes Ghost.
This change fixes that and adds logging for those errors.
closes https://github.com/TryGhost/Team/issues/1889
Portal's UI, specially for Signup and Account Upgrade, has changed a lot as we went from single tier -> multiple prices -> multiple tiers. This change removes the old unused `PlansSection` and related components that are no longer in use anywhere in Portal UI since we transitioned to Products(tiers).
- removes unused `ChangeProductPlansSection` component
- removes unused `PlanSectionStyles` to cleanup unused styles
refs https://github.com/TryGhost/Team/issues/2371
- playwright tests were broken due to state changes based on prev tests that were not accounted for
- in case of multiple newsletters, portal tests expected another step between stripe checkout for newsletter selection
- site settings test was disabling members, but not re-enabling it back
refs TryGhost/Team#2371
- check that members can unsubscribe from newsletters by toggling
preferences in their account settings
- check that member can log out
refs https://github.com/TryGhost/Team/issues/2376
When a discount offer for a first-payment is setup, when a member views their plan details in Portal, it still shows the discounted price, it shows the discounted offer instead of the actual price they will be charged next time, which is incorrect.
For one time payments, we'll not reflect the discount in portal as the member will not receive that discount going forward.
- tests were failing due to copy changes that made the test selectors fail to fetch the element
- for future, we should switch to using test ids here instead of text as they can easily change causing these failures
refs
f5aae1e2c5
refs
0f9ed54a6f
- changing playwright portal tests to work for single tier setup caused failure for comped upgrade tests as they were relying on button text that changed
refs https://github.com/TryGhost/Team/issues/2375
If a discount offer is associated with a tier that has a free trial enabled on full price / standard portal in membership settings, then the stripe checkout applied both the discount and free trial to the member, which is incorrect as we shouldn't be combining both.
- removes trial days from stripe checkout if a coupon is being applied, so only one of them is applied at a time
no issue
- moved logic from `<GhValidationStatusContainer>` to a new `validation-status` modifier
- removes a usage of the `ValidationState` mixin
- migrated uses of the component to a mixin
- paves the way for full removal of the `ValidationState` mixin in later refactors (mixins are deprecated)
- migrated `<GhFormGroup>` to a glimmer component
- swapped the extend of `GhValidationStatusContainer` to usage of the `validation-status` modifier with a template-only component
- updated all `<GhFormGroup>` to use the standard `class=""` instead of `@classNames=""` and `@class=""`
- allows `data-test-*` attributes to be added to uses of `<FormGroup>` to help when complex components are grouped as a form input
fixes https://github.com/TryGhost/Team/issues/2368
- Removed the usage of the `isLocalContentImage` Koenig util for the
email header and feature image url generation.
- While we were trying to set the width to 1200px, we didn't have that
size hardcoded. So that url would redirect back to the original location
instead of serving a smaller image. So I added a new internal size to
the `imageOptimization` config.
- This is fixed in both the new and old email flow and includes some
extra tests for the new flow.
refs. https://github.com/TryGhost/Team/issues/2371
- two extra assertion was needed for discount and free-trials to check
if the offers are listed in ‘Active’ offers and the URLs load portal
refs https://github.com/TryGhost/Team/issues/2371
- Test enabling private site and checking access with a password
- The test flow is lacking a check for site access through password
due to a Playwirght bug. This should be cleaned up in the future
refs. https://github.com/TryGhost/Team/issues/2371
- deleteAllMembers was an unnecessary step
- since there's a generated code appended to the name of the archived offer, it had to be shorter to avoid potential naming conflicts
refs https://github.com/TryGhost/Team/issues/2367
We're not releasing this immediately so need to put it behind the flag
so that we continue to send emails to the correct members.
refs https://github.com/TryGhost/Team/issues/2371
- Adds a test for publishing and sending
- Adds a test for email only sending
- Updated some util methods in the publishing spec to remove the dependency on the post bookmark (which is not present for email only posts)
refs https://github.com/TryGhost/Team/issues/2371
Note that the "Choose" button is "Continue" when running this test
standalone so currently it needs to run with the full suite.
refs https://github.com/TryGhost/Team/issues/2371
- extracting the re-used actions to utils allows tests to be self-descriptive rather than relying on comments and keeps the selectors and related actions in one place to help refactoring if/when they change
refs. https://github.com/TryGhost/Team/issues/2371
- Test for archived offers should be moved to ‘Archived’ view of the offer list in Admin, and the offer URL should redirect to the site's homepage for logged out visitors
refs https://github.com/TryGhost/Team/issues/2371
- bumped timeout between saving and refreshing to account for slower
speeds in CI
- increased specificity for the frontend text comparisons so the output
when failing is smaller and easier to parse
refs https://github.com/TryGhost/Team/issues/2371
- the 100ms timeout was enough for local tests to pass but was still failing on CI
- bumped to 200ms and skipped the creation of a new paragraph to reduce what the editor is doing
refs https://github.com/TryGhost/Team/issues/2371
- added timeout between clicking the editor and starting to type otherwise some of the typing events could be missed causing a mismatch in actual vs expected output
ref https://github.com/TryGhost/Team/issues/2371
- updated Member exports with csv validation
- added member fixtures to be loaded into Ghost to ensure filtering
works correctly when downloading / exporting members csv.
refs https://github.com/TryGhost/Team/issues/2371
- The "data-test-*" selectors in playwright did not work with publishing channel selectors. This is a quick hack to enable working around it
refs https://github.com/TryGhost/Team/issues/2367
This ensures that a Member is not considered subscribed to any emails, so that
counts for newsletter recipients are correct. Eventually we will filter members
on their email suppression status but this is not implemented yet.
Refs https://github.com/TryGhost/Team/issues/2371
- Tests whether the post access selection of public, members, or paid-members matches the expected post visibility on the frontend.
refs https://github.com/TryGhost/Team/issues/2371
- Adds a test that schedules a post 5 seconds in the future and waits
for it to be published
- Reduced the time restrictions for scheduling:
- The minimum time in the frontend is now 5 seconds in the future (came
from 5 minutes in the future)
- The time picker now suggests 10 minutes in the future instead of the
minimum scheduling time (came from 5 minutes)
- In the backend, a post will be allowed to be scheduled if it is at
least 2 minutes in the past (came from 2 minutes in the future)
- The scheduler will publish a post if it is at least 5 minutes in the
past, and maximum 5 minutes in the future (came from 2 minutes)
refs https://github.com/TryGhost/Team/issues/2371
- tests that a free member can upgrade to a paid tier via stripe checkout and the payment details are reflected in portal and member detail page on admin
refs https://github.com/TryGhost/Team/issues/2371
- tests modifying the content of a published post
- extracted publish flow into a `publishPost` function that returns a new browser page object with the newly created post loaded
refs https://github.com/TryGhost/Team/issues/2369
- this checks whether the Offer redemption count is set to 1, which
would be indicative that the Offer was successfully counted as
redeemed
no issue
- we want to run e2e tests against production Ghost instances and having test selectors available means much less brittle tests
- only real impact from keeping the selectors is extra file size in production builds but that is minimal (~6KB gzipped at time of commit)
refs https://github.com/TryGhost/Toolbox/issues/479
- this includes a handful of improvements to get Playwright working on a
local environment including:
- adding `testing-browser` environment so we don't nuke `development`
environments, and makes all the necessary changes to get Ghost to
behave when this is running
- stopped running one global instance of Ghost as this doesn't provide
a clean environment
- copies a few default fixtures that are needed for the new
environment
sentry ADMIN-C05
- resizing the window (or changing orientation) when viewing a single photo inside the Unsplash image selector was throwing errors because the event handler `setZoomedSize()` call was not passed the same arguments as the typical call made in `modify()`
- moved the `element` and `ratio` properties onto the class so they are preserved and ready to be used without being explicitly passed in when `setZoomedSize()` is called as part of an event
no issue
`<GhBillingIframe>` generates a request to the `/identities/` endpoint every time Admin is accessed for all users, however that endpoint is only accessible to users with the Owner role meaning we have a lot of unnecessary 403 errors in event logs and the developer console.
- added early exit when we know the logged in user doesn't have the Owner role
- removed the subscription fetching code that wasn't reachable (`token` was always `undefined`)
- the BMA sends subscription data as soon as it's available so the extra fetch isn't necessary
- we should start to keep tests grouped by their area, so first we split
by Admin tests and then Portal tests, and within that we split into
setup/Tiers/Offers etc
closes https://github.com/TryGhost/Team/issues/2242
Contributors don't have permission to fetch `/newsletters/` but the publish flow was sending a request every time a contributor opened a post in the editor creating noise in event logs and in the developer console.
- disabled the newsletters fetch when the logged in user is a contributor
- contributors can't publish so the "missing" data has no effect on the publish flow as it's not used
fixes https://github.com/TryGhost/Team/issues/2302
The analytics page should not be visible for Editors (and doesn't work currently anyway). This commit removes the button that goes to the analytics page for editors and authors.
refs https://github.com/TryGhost/Toolbox/issues/479
- e2e and integration test suites are running on port 2369. Playwright was not following this convention, without good reason.
- Port 2368 is the default port for development and production processes, so using it for test environment is not ideal
closes https://github.com/TryGhost/Team/issues/2241
- as part of the authenticated application setup, update the captured Sentry data with the user role
- helps narrow things down when we see permission errors pop up due to requests being made for endpoints that the current user doesn't have permission to access
refs: https://github.com/TryGhost/Toolbox/issues/481
* Correctly setup environment variable to run both local & staging browser-based tests
* Use non-production Ghost Admin build, since production builds require HTTPS to use Stripe Connect
refs https://ghost.slack.com/archives/C02G9E68C/p1670215917451249
When a member is deleted, and we receive an opened event for an email to
that member. We threw an uncaught Bookshelf EmptyResponse error.
- This change makes fetching the member not a requirement when handling
that event in the last seen at updater.
- It also adds try catches for all event listeners in the last seen at
updater
closes https://github.com/TryGhost/Team/issues/2275
When deleting a member, after confirming deletion another "unsaved changes" modal popped up. From that point, if you clicked to stay you remained on the member screen with stale data (the member was still deleted) resulting in further errors when any attempt to make changes was made.
- prevented the unsaved changes check running for a deleted member because it would always return `true` in that case
- ensured the data setup for the unsaved changes check still occurs when a member is accessed directly via the URL
- previously it was skipped because the data setup only occurred inside `fetchMemberTask` but that isn't called when the route already loaded the model via it's `model()` hook
refs https://github.com/TryGhost/Toolbox/issues/476
- The email verification trigger and host settings related bugs have been a cause of bugs in past releases. The admin client verification source did not have any test coverage in the past.
- The members test suite size is getting out of hand. This test is quite verbose, because of the state it's trying to check.
- In the future we should consider splitting up Member API (and probably other) test suites into smaller pieces.
no issue
- The sleep method has been used in 8 modules reimplementing the same thing over and over again. It's usually a sign of async event processing outside of the request/response loop. It's good to have a single point of implementation for a "hack" like this, so we could track it easier and address the even processing delay in a more optimal way centrally if it ever becomes a bottleneck
no issue
This will need some work, since we are introducing a 500ms delay to wait for a network request to return. Ideally the tier expander should eventually populate itself.
no issue
Local tests can now setup Stripe during the global setup process, and the webhook server is run out-of-process.
Running tests in CI against localhost will use environment variables to setup Stripe.
Providing a test URL will avoid setting up Stripe and will assume that it is already done.
no issue
This commit allows tests to run remotely by replacing selectors with production-suitable ones (no [data-test...]).
It also allows running locally with Stripe webhooks by adding a new global setup function.
refs. https://github.com/TryGhost/Team/issues/2327
- on the first two tabs of the email debug screen, the member email wasn't aligned properly when there was no name for a member
refs. https://github.com/TryGhost/Team/issues/2327
- Some minor CSS style changes were needed for errors on the email debug screen. Also we weren't showing any message i - for whatever reason - there was no data for batches.
refs. https://github.com/TryGhost/Team/issues/2327
- The length of the error messages for temporary and permanent failure tabs on the email debug screen can be arbitrary. This degrades scannability and limits the number of displayed rows in the list. Adding an expand button to the error message makes sense since the errors might repeat and the error code + the beginning of the message can be enough to understand them. Also this allows more rows to be displayed per screen.
refs https://github.com/TryGhost/Team/issues/2327
- wires email debug screen with real data from API
- fetches email batch data for showing all batches along with those errored
- fetches all recipient failures - temporary and permanent
- shows email settings that was used for sending out the email
fixes https://github.com/TryGhost/Team/issues/2346
- Adds email batch browse endpoint
- Adds email recipient failures browse endpoint
- Adds new fixtures and E2E tests for the new API
- Added support for snapshot tests to have 'nullable' types.
- Adds links and new modals for two new FAQs
- Has some checks for suppressions based on member property email_suppressions
- Using different technique for back buttons to simulate bigger stack
refs https://github.com/TryGhost/Team/issues/2348
closes https://github.com/TryGhost/Team/issues/2285
- added passthrough of `cursorDidExitAtTop` action so the editor can trigger title input focus on key commands that trigger the cursor to leave the top of document
refs https://github.com/TryGhost/Team/issues/2255
These methods will be used by the Mailgun implementation of EmailSuppressionList
so that emails are removed from both our internal list and Mailguns.
refs https://github.com/TryGhost/Team/issues/2339
- Includes a new pattern in the job manager that allows us to properly
await jobs.
- Added new convenience mocking methods to stub settings
- Tests the main flows for bulk sending:
- Sending in multiple batches
- Sending to multiple segments
- Handling a failed batch and retrying that batch
- Fixes bug in batch generation (ordering not working)
In a different PR I'll add more detailed tests.
refs https://github.com/TryGhost/Team/issues/2327
As part of improving visibility for email failures, this change adds a new debug screen that allows visualising the email failures for a post. The screen is hidden on the UI for now and only accessible via URL directly.
Co-authored-by: Djordje Vlaisavljevic <dzvlais@gmail.com>
We can fetch the same event multiple times from Mailgun so we need to
be able to protect against inserting duplicate events in the
database. This will allow us to catch duplicate errors on insert when
handling complaint events.
no issue
- This module is marked for deletion with a todo comment. It also popped up as least covered place in the codebase, because it was never used.
fixes https://github.com/TryGhost/Team/issues/2332
Saves events in the database and collects error information.
Do note that we can emit the same events multiple times, and as a result
out of order. That means we should correctly handle that a delivered
event might be fired after a permanent failure. So a delivered event is
ignored if the email is already marked as failed. Also delivered_at is
reset to null when we receive a permanent failure.
closes https://github.com/TryGhost/Team/issues/2286
- use the `registerAPI` prop to get access to a basic API for focusing and inserting paragraphs
- replaced commented mobiledoc based title key handling with lexical handling
closes https://github.com/TryGhost/Team/issues/2309
- adds new mailgun provider to send out batch emails
- updates sending service to send email id for mailgun provider, allows tagging mail with email id
refs a2d487e074
- Same reasoning as in referenced commit: "Database schema definition file is a special type of "configuration" file containing mostly static declarations. This sort of code should not be tested by unit tests, rather by e2e tests"
refs https://github.com/TryGhost/Toolbox/issues/486
- Frontend helpers are extremely hard to cover fully in e2e tests and are better suited to be covered by unit tests (which they have very hight coverage with)
closes https://github.com/TryGhost/Team/issues/2324
- It seemed like the "limit" query parameter did not work properly returning multiple entries from the endpoint. In reality the whole query string was ignored because of an error in the "filter" part of the query ^_^
refs https://github.com/TryGhost/Team/issues/2326
- The job takes considerably longer to run with MySQL, so needed a longer sleep time. It's a temporary fix to unblock a broken build. We should investigate why the job takes so long to run on MySQL
refs fc9f8aebc1
- With integration & regression test suites included in the e2e coverage reports the coverage has jumped up nicely. We should keep the threshold as hight as possible from accidentally reducing the coverage quality
refs https://github.com/TryGhost/Team/issues/2225
- updated the `formatOnWrite` transform map for posts to include the new `nodes` and `transformMap` options used by `urlUtils` for transforming node payload data
- added `nodes` to the `lexicalLib` module that pulls in our default nodes to be passed in to the URL transform utilities
- added `urlTransformMap` to the `lexicalLib` module that maps transform type and data type to URL transform utility functions that accept a single URL argument
refs https://github.com/TryGhost/Team/issues/2317
This table is used for persisting the email suppression list.
We don't have a member_id column because emails, not members are suppressed.
fixes https://github.com/TryGhost/Team/issues/2308
- Still has some missing pieces, but mostly works.
- Uses new handlebars template for emails
- When sending emails with the new email stability flag enabled, one
test email is now sent via the default smtp ghost mailer.
fixes https://github.com/TryGhost/Team/issues/2310
This moves the processing of the events from the event-processor to a
new email-event-processor in the email-service package.
- The `EmailEventProcessor` only translates events from
providerId/emailId to their known emailId, memberId and recipientId, and
dispatches the corresponding events.
- Since `EmailEventProcessor` runs in a separate worker thread, we can't
listen for the dispatched events on the main thread. To accomplish this
communication, the events dispatched from the `EmailEventProcessor`
class are 'posted' via the postMessage method and redispatched on the
main thread.
- A new `EmailEventStorage` class reacts to the email events and stores
it in the database. This code mostly corresponds to the (now deleted)
subclass of the old `EmailEventProcessor`
- Updating a members last_seen_at timestamp has moved to the
lastSeenAtUpdater.
- Email events no longer store `ObjectID` because these are not
encodable across threads via postMessage
- Includes new E2E tests that test the storage of all supported Mailgun
events. Note that in these tests we run the processing on the main
thread instead of on a separate thread (couldn't do this because
stubbing is not possible across threads)
There are some missing pieces that will get added in later PRs (this PR
focuses on porting the existing functionality):
- Handling temporary failures/bounces
- Capturing the error messages of bounce events
refs https://github.com/TryGhost/Team/issues/2291
When sending out mails to individual recipients, its possible that recipient gets a temporary or permanent failure for receiving the mail. Temporary failures can generally get resolved after a bit when the recipient’s mail server accepts the email, unlike permanent failures. For both customer visibility and easier debugging on what went wrong while delivering to a particular recipient, we’ll store the permanent/temporary failure for a recipient.
- migration adds a new table that stores the failure information for the recipients
refs https://github.com/TryGhost/Toolbox/issues/475
refs https://github.com/TryGhost/Toolbox/issues/117
- The frontend/src folded would never get picked up by a code coverage tooling as scripts there are dynamically minimized and served from the server
- There's nothing to cover under frontend/public
- Logic behind these changes is the same as in second referenced issue
fixes https://github.com/TryGhost/Team/issues/2326
When importing more than 500 members, we didn't testImportThreshold at
the right time. It was called too early because the importing job was
not awaited. This also adds an E2E test for this case.
closesTryGhost/Team#2313
- Added Sent event to Post analytics and Members feed. Now post can be
Sent or Received or Bounced.
- Excluded Delivered event from Sent filter on backend.
refs https://github.com/TryGhost/Ghost/security/advisories/GHSA-9gh8-wp53-ccc6
refs https://github.com/TryGhost/Toolbox/issues/465
- Bookshelf relations allows us to edit relational records by default, which was used liberally in the codebase.
- Not having a clear track record of editable relations left the model layer prone to triggering unwanted nested saves and created a vulnerability where members were able to edit newsletter settings.
- With explicit editable relations it's easier to keep track of relations having editable access to related records. Makes the relational data modification pattern safer to use too.
- Anyone running 5.x should update to 5.24.1
Credits: Dave McDaniel and other members of [Cisco Talos](https://talosintelligence.com/vulnerability_reports)
refs https://github.com/TryGhost/Ghost/security/advisories/GHSA-9gh8-wp53-ccc6
refs https://github.com/TryGhost/Toolbox/issues/465
- Bookshelf relations allows us to edit relational records by default, which was used liberally in the codebase.
- Not having a clear track record of editable relations left the model layer prone to triggering unwanted nested saves and created a vulnerability where members were able to edit newsletter settings.
- With explicit editable relations it's easier to keep track of relations having editable access to related records. Makes the relational data modification pattern safer to use too.
- Anyone running 5.x should update to 5.24.1
Credits: Dave McDaniel and other members of [Cisco Talos](https://talosintelligence.com/vulnerability_reports)
closes Sentry ADMIN-CC8
closes Sentry ADMIN-DDM
closes Sentry ADMIN-C9F
- there are situations where the editor range when key commands are triggered does not have a head section which then throws errors due to the commands assuming there will always be a section present
- added a guard to key commands that use the head section to abort early and fall back to default Mobiledoc behaviour when the section is missing
closes sentry ADMIN-C7S
- we can't guarantee that the iframe being swapped to has rendered yet at the time we swap iframes so we need a guard around the `contentDocument.body` property existing before setting it's `scrollTop` value
refs https://github.com/TryGhost/Team/issues/2280
refs 9b0c21e0a2
As part of the email stability work, we added new `source` and `source_type` columns to `email` table, which allows us to store the email source information. The source for all existing emails before the stability work was always `html`, while newer emails will store `mobiledoc` or `lexical` directly.
While the `source` for all existing emails was populated with the `html` as part of above migration, we also need to store the right `source` for all new emails created till the feature is under a flag.
This change updates the current email flow to also store `source` with html data, so it can be used in future with new email service and allows removing old `html` column.
closesTryGhost/Team#2264
- Instead of relative paths, we can use absolute. It helps to keep code cleaner and don't worry about import when doing refactoring.
Relative paths require rewriting them in case a file is moved to another directory.
refs https://github.com/TryGhost/Team/issues/2253
refs https://github.com/TryGhost/Team/issues/2254
This package is analogous to the @tryghost/member-events package. The
events here will be consumed by the EmailSuppressionList
implementation and used to add emails to said list. They'll be
dispatched by the code which handles events received from Mailgun.
refs: https://github.com/TryGhost/Team/issues/1121
- We've decided on one preferred message for unexpected errors
- We want to use this everywhere where we don't know what to display
- We now have a GENERIC_ERROR_MESSAGE constant that we should use
refs: https://github.com/TryGhost/Team/issues/1121
- showAPIError is a method intended for formatting errors from the Ghost API
- Ghost API Errors do not have a detail field, therefore this code was redundant
- there are also no related tests
- removing now because I'm trying to cleanup and streamline all our error handling code
refs: https://github.com/TryGhost/Team/issues/1121
- In certain cases our API sends the same data for message and context.
- We will also fix this server-side, but we should also be smart in the UI and not show duplicate info
no issue
- "unhandled" `TransitionAborted` errors almost always occur as part of expected application behaviour and were causing a lot of noise in Sentry making it harder to track down real errors
- when a `TransitionAborted` error occurs outside of expected behaviour it will usually be accompanied by other errors that do get logged
- there's a long-standing Ember issue about how aborted transition errors should be handled at https://github.com/emberjs/ember.js/issues/12505
refs: https://github.com/TryGhost/Team/issues/1121
refs: 54574025e0
- The previous change to fall back to a generic error on the server side is resulting in lots of much less useful Sentry reports
- For unexpected errors, change what's sent to Sentry back to context
- This is done by adding a specific code, so we don't have to match on a string that might change
- Also add the error type, id, code & statusCode as tags to the events - these are searchable structured data
- Adding code as a tag also makes it possible to find all errors that showed the generic message
- As demonstrated by my comments in the boot file, I thought sentry was already depending on the version package
- IMO it's undesirable to require package.json directly esp when we have a tool setup and ready for tis
- Added a bunch of tests to show that Sentry does roughly what we think
fixes https://github.com/TryGhost/Team/issues/2284
New batch sending flow (still WIP). Logs the sent emails instead of actually sending them. Unit tests are coming in later commits.
refs https://github.com/TryGhost/Team/issues/2280
We are moving away from storing html and plaintext on email and instead will store the email data in source and source_type columns which allows us to store the email in other formats like mobiledoc and lexical. Storing in those formats allows greater flexibility for later html generation
- adds new `source` column that stores `mobiledoc`/`lexical`/`html` data for a newsletter
- adds new `source_type` column that stores one of `mobiledoc`/`lexical`/`html` to identify type of source
closes https://github.com/TryGhost/Team/issues/2290
Currently, if the whole batch of email fails to send we don’t capture
any errors directly tied to the batch. This makes it hard to debug which
and why a batch failed when debugging email errors. Going forward we'll
store the error information for a failing email batch directly that
allows easier debugging for batch.
- `error_status_code` : Captures statusCode returned by Mailgun,
available in error.status from the example batch error
- `error_message` : Captures short error message from Mailgun and
status, available in context object of batch error
- `error_data` : Captures while whole error json for a batch. As
mentioned in pitch, this will be huge data and we’ll figure out long
term how to best use this.
refs: https://github.com/TryGhost/Team/issues/2289
refs: https://github.com/TryGhost/express-hbs/issues/161
- Themes that resuse layouts as templates trigger horrible errors, which are thrown as 500s
- But there's nothing the server is doing wrong, it's a theme user, so we downgrade these to 400s
- There is more to do here to improve the errors shown, but this is just a first step to ensure that theme issues don't look like server failures
refs: https://github.com/TryGhost/Toolbox/issues/479
Framework includes:
* command to run tests
* command to record tests
* mechanism for starting and stopping Ghost before and after each suite of tests
* mechanism for loading fixtures into Ghost before starting tests
* sample test for controlling Ghost Admin
closes https://github.com/TryGhost/Team/issues/2295
Problem:
- `<GhUploader>` is not yet converted to an Octane component so it's arguments are not read-only
- when a file is selected it sets it's `files` property which in turn updates the tracked `files` property that was passed in, and then again updates it to an empty file list when the input field is cleared
- that tracked property was never cleared once the product image was uploaded resulting in a "re-upload" attempt with an empty file list every time the product card was put back into edit mode
Fix:
- added a guard in `<GhUploader>` so it doesn't try to upload an empty file list if one is passed in as an attribute
- added a reset of the tracked `files` property in the product card once the image upload is complete
no issue
- The name of the "StaffService" is ambiguous and too generic. Lack of good naming makes one to dig into the implementation details figuring out what the service does.
- Should be named a more descriptive way
refs: https://github.com/TryGhost/Team/issues/1121
refs: dfffa309a8
- This makes a fundamental change to Ghost's server side error handling, so that no unhandled errors are used as API responses
- Anything that has been handled and rethrown as a Ghost error cna be trusted
- We also already trust a couple of known errors from bookshelf and handlebars
- Everything else is assumed to be a code error, and should not be shown as the main message
- Instead we use our generic fallback message and use the OG error as context
fixes https://github.com/TryGhost/Team/issues/2282
Added a new email service package that is used when the email stability
flag is enabled. Currently not yet implemented so will throw an error
for all entry points (if flag enabled).
Removed usage of `labs.isSet.bind` across the code, because that breaks
the stubbing of labs by `mockManager.mockLabsEnabled` and
`mockManager.mockLabsDisabled`. `flag => labs.isSet(flag)` should be
used instead.
All email depending tests now disable the `emailStability` feature flag
to keep the tests passing + make sure we still run all the tests for the
old flow while the email stability package is being built.
refs https://github.com/TryGhost/Team/issues/2268
The approach of using the service to lead email suppression data as
opposed to bookshelf relations allows us to wire things up without
having implemented the database. The getBulkSuppressionData allows us to
do this without much of a DB performance hit.
closes https://github.com/TryGhost/Team/issues/2126
- Cleaned up the following GA flags: `newsletterPaywall`, `freeTrial`, `compExpiring`, `searchHelper`, `emailAlerts`, `fixNewsletterLinks`.
refs https://github.com/TryGhost/Team/issues/2276
Portal died with an unintelligible error about portal plans/includes being undefined when there was another react error on the page. The other error somehow caused the `site` data to be empty object for Portal, and there was one place where we didn't check for the empty object while transforming the site data which caused the failure.
This change adds the guard for failing check so we don't throw the error in such cases and process as normal.
refs https://github.com/TryGhost/Team/issues/2267
This will eventually be replaced by a Mailgun specific implementation,
but for now we're using mock one which responds with fake data for
local development.
refs https://github.com/TryGhost/Team/issues/2267
This package contains definitions for the public interface of the
email suppression list as well as an abstract implementation.
refs: https://github.com/TryGhost/Team/issues/1121
- This makes several key changes to the way errors are handled in the member importer, to ensure that we only show error messages to users that we wrote.
- Fundamentally, we no longer trust all API errors, and instead only trust a set of very specific API errors. Anything outside of that is replaced with a generic error message.
- Also switches the server-side error generated for email verification (which can throw during member import) to be a HostLimitError, as that is a more appropriate class.
- Note: there are many other parts of Ghost admin that need a similar overhaul, and a similar change we need to introduce server side to fully resolve the underlying issue of bubbling up code errors to the UI.
