Commit Graph

518 Commits

Author SHA1 Message Date
Fabien "egg" O'Carroll
49af26279d Updated MailgunEmailSuppressionList to only handle previous bounces
We're seeing behaviour from Mailgun where permanent failures with a
5xx error code are not being added to their internal suppression list,
which is resulting in the Ghost list becoming out of sync with
Mailgun.

Rather than adding emails to the suppression list when Mailgun does,
we're instead going to add emails _after_ Mailgun does, by waiting for
an error code which tells us the email is already on the suppression
list.

Those codes are 605 for previous bounces and 607 for previous spam complaints.
2023-01-11 15:10:39 +07:00
Elena Baidakova
9c664c8d02
Added data attributes to admin/tiers (#16095)
refs TryGhost/Team#2371
2023-01-11 11:35:55 +04:00
Simon Backx
117afb232c Improved stability of batch sending test
refs https://github.com/TryGhost/Ghost/actions/runs/3884901582/jobs/6628075997

The 'Doesn't include members created after the email in the batches' test sometimes failed randomly because of timing. This change ensures a strict timing.
2023-01-10 17:12:56 +01:00
Simon Backx
9ca2e3f183 🐛 Fixed storing email recipient failures
fixes https://github.com/TryGhost/Team/issues/2398

There was an error when fetching the existing email recipient failure. It ended up matching all recipient failures. The result was that only one failure was stored in the database.
2023-01-10 15:41:42 +01:00
Rishabh
7a74ec8daa Updated playwright tests to use updated data ids
refs https://github.com/TryGhost/Team/issues/2371

- cleans up and adds comments for portal playwright tests
- updates data test attributes for portal trigger and popup selectors for consistency
- updates data attribute usage for offers
2023-01-10 13:33:12 +05:30
Rishabh
628b80778f Updated create tier playwright util for all archived tiers
refs https://github.com/TryGhost/Team/issues/2371

- in case all tiers are archived before new tier is created, the add tier section can be collapsed and will need to be opened first before going through add tier flow
2023-01-10 09:34:21 +05:30
Simon Backx
26635f192f Added visible theme errors in admin
fixes https://github.com/TryGhost/Team/issues/2393
2023-01-06 15:51:34 +01:00
Simon Backx
f6e463ea43 Fixed missing awaits in publishing Playwright test
no issue
2023-01-06 14:18:54 +01:00
Rishabh
54b27f7d55 Fixed offers playwright test for redemption count
- updates check of redemption count using data test attributes to avoid unreliable member count check
2023-01-06 18:39:38 +05:30
Simon Backx
7b3712a15b
Added visible theme errors in admin (#16081)
refs https://github.com/TryGhost/Team/issues/2393

- During boot and loading the active theme, we now cache the result of
the gscan validation. Cache configuration can happen in
`adapters.cache.gscan`
- We now also return non-fatal errors when activating or adding a theme.
- When the `themeErrorsNotification` feature flag is on, we fetch the
active theme (which includes the validation information) when loading
admin
- If the currently active theme has errors, we show an error
notification that can open the error modal
- Added a new endpoint: `/ghost/api/admin/themes/active/` that returns
the result of the last gscan validation of the active theme. If no cache
is available, it will run a new gscan validation.
- Added new permissions for the active action/endpoint (author, editor,
administrator)
2023-01-06 13:44:27 +01:00
Naz
3e209218d7
Made resource mismatch error more specific
refs https://github.com/TryGhost/Toolbox/issues/497

- During gscan fatal error downgrade to non-fatal some of the deprecated helpers were a bit vague to debug with no information on which exact "resource" was invalid
- Added resource name to the log for clarity. Should make life easier when debugging potential get helper misuses
2023-01-06 18:05:07 +07:00
Sam Lord
36e7e79eec Improved Playwright test, selector for redemption count
no issue

Selector for redemption count was not ideal - this makes it more specific & clear.
2023-01-06 10:56:19 +00:00
Rishabh
30c9dfd68d Fixed failing playwright test to remove member labels
- the test was using incorrect test state that was copied over from adding label test
- also adds guard for empty newsletters in member filters as in some cases it might not exist as found by test
2023-01-06 16:17:34 +05:30
Ronald Langeveld
e52f29231f Added Playwright test - remove labels from members
ref https://github.com/TryGhost/Team/issues/2371

- check that a fitered list of members can have a label removed from
  them at once.
2023-01-06 14:36:13 +08:00
Fabien "egg" O'Carroll
953f3856a8 Handled EmailBounceEvent with 605 error code
When Mailgun fails to deliver an email to an address because the
address has already bounced before, it gives us a permanent fail event
with a 605 error code rather than a 5xx one. Because we want to
"backfill" our suppressions data with previously bounced email
addresses, we want to handle this specific error code.

We may update this logic in the future based on new information from
Mailgun with respect to their 6xx error codes and the
meanings/underlying cause of theme.

This also moves the tests which check for whether or not emails are
suppressed into their own fail so that we do not pollute the event
storage tests, and adds more tests cases.

We also fix a leaky sinon stub which we were not resetting in the email
event storage tests
2023-01-05 17:11:37 +07:00
Fabien "egg" O'Carroll
2d9114450c Fixed email_recipients fixtures for tests
The email_recipient fixtures were using duplicate and mismatched email addresses
rather than having them correctly map to the Members, which is required for testing
email suppressions.
2023-01-05 17:11:06 +07:00
Simon Backx
913ad18b71
Added DomainEvents.allSettled utility method (#16075)
no issue

With the increased usage of DomainEvents, it gets harder to build
reliable tests without having to resort to timeouts. This utility method
allows us to wait for all events to be processed before continuing with
the test.

This change should speed up tests and make them more reliable.

It only adds extra code when running tests and shouldn't impact
production.
2023-01-04 14:30:35 +01:00
Rishabh
f5aa07a095 🐛 Removed free trial message shown on portal for invite only sites
closes https://github.com/TryGhost/Team/issues/2361

If a free trial tier existed on site and its set to 'Invite only' in membership settings, the free trial copy still showed on portal.

- removes free trial copy from portal if site is invite only
- adds playwright test to make sure free trial copy is not shown for invite only sites
2023-01-04 17:11:55 +05:30
Fabien 'egg' O'Carroll
50e99e013c
Added migrations to drop and recreate the suppressions table (#16070)
There are currently two issues with the suppressions table:
  - We have some incorrect rows
  - We have missing UNIQUE constraints

We want to completely wipe the tables and start fresh, as well as make
sure that the UNIQUE constraints are added, so we drop the table
completely, and then re-add it, which should result in an empty
suppressions table with all expected constraints.

We've also renamed the `email_address` column to `email` to match our
`users` & `members` tables
2023-01-04 17:26:57 +07:00
Simon Backx
819d0d884c
Improved email verification required checks (#16060)
fixes https://github.com/TryGhost/Team/issues/2366
refs https://ghost.slack.com/archives/C02G9E68C/p1670232405014209

Probem described in issue.

In the old MEGA flow:
- The `email_verification_required` check is now repeated inside the job

In the new email service flow:
- The `email_verification_required` is now checked (didn't happen
before)
- When generating the email batch recipients, we only include members
that were created before the email was created. That way it is
impossible to avoid limit checks by inserting new members between
creating an email and sending an email.
- We don't need to repeat the check inside the job because of the above
changes

Improved handling of large imports:
- When checking `email_verification_required`, we now also check if the
import threshold is reached (a new method is introduced in
vertificationTrigger specifically for this usage). If it is, we start
the verification progress. This is required for long running imports
that only check the verification threshold at the very end.
- This change increases the concurrency of fastq to 3 (refs
https://ghost.slack.com/archives/C02G9E68C/p1670232405014209). So when
running a long import, it is now possible to send emails without having
to wait for the import. Above change makes sure it is not possible to
get around the verification limits.

Refactoring:
- Removed the need to use `updateVerificationTrigger` by making
thresholds getters instead of fixed variables.
- Improved awaiting of members import job in regression test
2023-01-04 11:22:12 +01:00
Fabien 'egg' O'Carroll
e78612bb66
Fixed MailgunEmailSuppressionList adding non-5xx failures to the list
The MailgunEmailSuppression list was incorrectly adding emails
to the suppression list for permanent failure events which have
an error code outside of the 5xx range.
2023-01-04 17:03:52 +07:00
Simon Backx
789e2c96c0
🐛 Fixed SingleUseTokens being cleared on boot (#15999)
fixes https://github.com/TryGhost/Team/issues/1996

**Issue**
Our Magic links are valid for 24 hours. After first usage, the token
lives for a further 10 minutes, so that in the case of email servers or
clients that "visit" links, the token can still be used.

The implementation of the 10 minute window uses setTimeout, meaning if
the process is interrupted, the 10 minute window is ignored completely,
and the token will continue to live for the remainder of it's 24 hour
validity period. To prevent that, the tokens are cleared on boot at the
moment.

**Solution**

To remove the boot clearing logic, we need to make sure the tokens are
only valid for 10 minutes after first use even during restarts.

This commit adds 3 new fields to the SingleUseToken model:
- updated_at: for storing the last time the token was changed/used). Not
really used atm.
- first_used_at: for storing the first time the token was used
- used_count: for storing the number of times the token has been used

Using these fields:
- A token can only be used 3 times
- A token is only valid for 10 minutes after first use, even if the
server restarts in between
- A token is only valid for 24 hours after creation (not changed)

We now also delete expired tokens in a separate job instead of on boot /
in a timeout.
2023-01-04 09:49:39 +01:00
Kevin Ansfield
0b80a7f9eb
Fixed new tier creation browser tests (#16066)
refs https://github.com/TryGhost/Ghost/pull/16013

- ordering of tiers changed so newly added tiers are no longer always at the end of the list in Admin
2023-01-03 17:59:33 +00:00
Simon Backx
803bb18b8d Improved unit test coverage for the output mappers
no issue

Increases the unit test coverage to a bit above 60% to increase margin
2023-01-03 15:58:31 +01:00
Elena Baidakova
235446b034
🐛 Fixed tiers order by monthly_price (#16013)
refs TryGhost/Team#2273
- According to documentation (https://ghost.org/docs/content-api/#order)
tiers should be ordered from lowest to highest monthly price.
2023-01-03 12:59:18 +04:00
Simon Backx
f568b35f26 Fixed 2023 snapshot tests
no issue

Updated the snapshot files for the email preview test snapshots that contain 2023 in the email footer.
2023-01-02 11:18:07 +01:00
Rishabh Garg
2eac41b1f4
🐛 Fixed invalid email getting saved for members (#16021)
refs https://github.com/TryGhost/Team/issues/2235

We found some cases which can cause a site to have member emails that have invalid characters like `member@example.com�`. This happened due to the `validator` version used by Ghost not able to catch some specific cases as invalid email, allowing members to be created with them either via Admin or Importer or direct signup. Portal UI already blocked these email as invalid. This change:

- updates `@tryghost/validator` to include a latest version of email validator that catches these invalid cases
- doesn't allow member creation with invalid email like above
- doesn't allow existing member emails to be edited to invalid
2022-12-16 16:47:52 +05:30
Hannah Wolfe
5f90baf6fe
Added Revue Importer (#16012)
refs: https://www.getrevue.co/app/offboard

- Revue is stopping all paid subscriptions on 20th Dec, and shutting down on Jan 18th.
- This update allows Ghost to accept and handle the zip file Revue are providing as an export in Labs > Importer
- It will import posts (as best as we can with the data provided) and subscribers as free members
- At present it doesn't import paid subscribers, as we don't have that info, but you can disconnect Revue from your Stripe account to prevent all your subscriptions being cancelled & there's the option this can be fixed later
- There will be further updates to polish up this tooling - this is just a first pass to try to get something in people's hands

Co-authored-by: Paul Davis <PaulAdamDavis@users.noreply.github.com>
2022-12-15 17:22:54 +00:00
Simon Backx
0825a2d7f4
🐛 Fixed importing existing member resetting newsletters (#16017)
fixes https://github.com/TryGhost/Team/issues/2386

**Issue:**
- When trying to import a member that already exists, and has
'subscribed' set to 'true' in the CSV, the newsletters the member is
subscribed to are reset to the default newsletters.
- When ediging a member with the API and setting `subscribed` to true,
the same happens.

**Cause:**
A faulty check for the `status` property of a newsletter.

Fixed and added a new E2E test.
2022-12-15 17:45:11 +01:00
Hannah Wolfe
5fad27a3c0
Added html -> mobiledoc conversion to the importer (#16016)
- Now that the importer runs in a job, it seems sensble that we should
do this
- If posts are imported with HTML set, but not mobiledoc, we now convert html -> mobiledoc
- Note: This also converts the mobiledoc -> html so _may_ be lossy
- Without this, imports that only have HTML, not mobiledoc, would have
resulted in empty posts, so lossy > empty
2022-12-15 16:10:36 +00:00
Simon Backx
94e85dc09e
Reduced webhook calls when updating last_seen_at for email opens (#16008)
refs https://ghost.slack.com/archives/C02G9E68C/p1670960248186789

This reverts a change that was made here:

f4fdb4fa6c (r93071549),
but it still moved the original code to a new location in the
LastSeenAtUpdater

It includes a new E2E test to make sure timezones are supported
correctly.

- By not using Bookshelf, we no longer fire webhook calls
- By not using the member repository, we don't fetch and update the
member model and the labels relation in a forUpdate transaction, which
caused deadlock issues on the labels/members_labels tables which were
hard to resolve. Until now I was unable to find the other conflicting
transaction that caused this deadlock. Moving to raw knex (instead of
Bookshelf) and only updating the last_updated_at column should remove
the deadlock issue.

This removed the test for the email service wrapper, since it started
failing for an unknown reason and the test didn't make much sense (was
added earlier only to bump test threshold).
2022-12-14 17:50:42 +01:00
Hannah Wolfe
270f288c48 Added a timeout to the get helper
- The get helper can sometimes take a long time, and in themes that have many get helpers, the request can take far too long to respond
- This adds a timeout to the get helper, so that the page render doesn't block forever
- This won't abort the request to the DB, but instead just means the page will render sooner, and without the get block
2022-12-14 15:35:07 +00:00
Kevin Ansfield
a69ffb94c1 Added playwright test for publishing posts for specific tiers
refs https://github.com/TryGhost/Team/issues/2371

- test publishes a post with access for a single tier then checks the front-end with no member, member on wrong tier, and member on right tier
2022-12-14 12:20:55 +00:00
Rishabh
25c530293f Added playwright tests for forever and multiple month offers
- adds test that cover creating and signing up to multiple-month/forever offers
- checks that the offer information is shown to members during signup and in account detail
2022-12-14 17:13:20 +05:30
Rishabh
199d8644a5 Fixed offers playwright test for one-time discount
refs aa693039a3

- one-time offer discounts are no longer shown to member in Portal settings, as portal shows only the next payment info.
2022-12-14 16:27:01 +05:30
Simon Backx
47cd7a7095
🐛 Handled unknown Mailgun events (#15995)
refs https://ghost.slack.com/archives/C02G9E68C/p1670916538764019

- We receive events that don't have an emailId or providerId.
- We filter those events now and log them as an error
2022-12-14 11:17:45 +01:00
Rishabh
aa693039a3 🐛 Removed expired offers shown in portal account detail
closes https://github.com/TryGhost/Team/issues/2376#event-8026429598

- if an offer is expired/in past, we no longer show it in member account info against the price
- one-time offers are never showed in portal in member account detail, as the payment information shown to member in Portal points to charge at next payment
- if trial days are over for a subscription, portal doesn't show any offer data on member account detail
2022-12-14 15:27:49 +05:30
Elena Baidakova
9579791185
Resubscribe member to default newsletters after removing from suppression list (#15974)
refs TryGhost/Team#2372
2022-12-13 13:31:50 +04:00
Elena Baidakova
7ef157b17f
Improved count of items in e2e tests (#15986)
refs TryGhost/Team#2371
- We need to wait selectors before count them
https://github.com/microsoft/playwright/issues/14278

These changes needed to make the results of tests more stable on the CI
2022-12-12 16:51:31 +04:00
Rishabh
ce45571dc0 Updated email faqs in portal to show sender email address
refs https://github.com/TryGhost/Team/issues/2348

- updates email faqs to show the sender email from newsletter data
2022-12-12 15:09:32 +05:30
Rishabh
4e766e2584 Fixed broken playwright tests
refs https://github.com/TryGhost/Team/issues/2371

- playwright tests were broken due to state changes based on prev tests that were not accounted for
- in case of multiple newsletters, portal tests expected another step between stripe checkout for newsletter selection
- site settings test was disabling members, but not re-enabling it back
2022-12-12 14:50:34 +05:30
Elena Baidakova
88b056dc86
Added Playwright member's actions tests (#15968)
refs TryGhost/Team#2371
- check that members can unsubscribe from newsletters by toggling
preferences in their account settings
- check that member can log out
2022-12-12 10:13:08 +04:00
Rishabh Garg
109bce8096
Fixed portal comp upgrade playwright tests (#15977)
refs
f5aae1e2c5
refs
0f9ed54a6f

- changing playwright portal tests to work for single tier setup caused failure for comped upgrade tests as they were relying on button text that changed
2022-12-09 19:19:13 +05:30
Peter Zimon
527d718b76
Added extra assertions to Offers tests (#15956)
refs. https://github.com/TryGhost/Team/issues/2371

- two extra assertion was needed for discount and free-trials to check
if the offers are listed in ‘Active’ offers and the URLs load portal
2022-12-09 08:56:41 +01:00
Naz
ffd07ea744
Added Playwright "Private site" test
refs https://github.com/TryGhost/Team/issues/2371

- Test enabling private site and checking access with a password
- The test flow is lacking a check for site access through password
due to a Playwirght bug. This should be cleaned up in the future
2022-12-09 14:01:19 +07:00
Peter Zimon
be84593684 Fixed offers archive test
refs. https://github.com/TryGhost/Team/issues/2371

- deleteAllMembers was an unnecessary step
- since there's a generated code appended to the name of the archived offer, it had to be shorter to avoid potential naming conflicts
2022-12-09 07:48:24 +01:00
Ronald Langeveld
d62f09c588 Added filtered list labelling playwright test
ref https://github.com/TryGhost/Team/issues/2371

- Filters and then adds label to filtered members
2022-12-09 12:56:34 +07:00
Simon Backx
e7572fc111 Added Playwright test for disabled subscription access
refs https://github.com/TryGhost/Team/issues/2371

Portal should be disabled on the frontend, sending as a newsletter should no longer be an option when publishing
2022-12-08 17:58:03 +01:00
Djordje Vlaisavljevic
82aa9522dc Added playwright test for updating atier
refs TryGhost/Team#2371

- Tests that tier details can be updated
- Tests that the updates are reflected in Portal
2022-12-08 17:40:42 +01:00
Simon Backx
6af149120a Added Playwright test for invite only sites
refs https://github.com/TryGhost/Team/issues/2371

Tests if an invite only site disables sign up.
2022-12-08 17:27:30 +01:00