Commit Graph

56 Commits

Author SHA1 Message Date
Simon Backx
21ded0b4b8 Corrected email link in subscription box
refs https://github.com/TryGhost/Team/issues/2674

The replacement didn't work inside an attribute, so I removed it to a link that doesn't do anything.
2023-03-24 10:23:51 +01:00
Simon Backx
a9791e95cc Fixed including current email in last 3 posts in newsletters
refs https://github.com/TryGhost/Team/issues/2675

The current post could be inclnuded in the last 3 posts if it was also published. This is now fixed and covered by a test.
2023-03-24 09:20:32 +01:00
Simon Backx
0107d2bb77 Fixed subscription status not showing correctly in emails
refs https://github.com/TryGhost/Team/issues/2674

- The segment detection doesn't work outside the main post content. So the data-gh-segment attribute didn't work. It is now replaced with just a simple email replacement that is empty for a free member.
- Fixed that a trialing member was shown as 'paid'. This is now replaced with 'trialing'.

This commit also includes E2E tests for a couple of member statusses.
2023-03-24 08:55:36 +01:00
Sanne de Vries
3f0ec1af3d Updated subscription box in email preview and email template
Refs https://github.com/TryGhost/Team/issues/2738

- With all footer additions and changes, the subscription box did not match the rest of the design and is therefore updated.
2023-03-23 19:39:58 +01:00
Sanne de Vries
0455672832 Updatet latest post section in email template
Refs https://github.com/TryGhost/Team/issues/2675
2023-03-23 13:16:48 +01:00
Daniel Lockyer
2941154353
Added dynamic date matching to email sending snapshot
- we don't want to include the actual date here because it'll change
- this adds a regex to match the date and replace it with a standard
  "date" string, which won't change
2023-03-23 09:20:23 +01:00
Simon Backx
860048a89d Fixed snapshots 2023-03-22 17:03:11 +01:00
Simon Backx
55c281debf Reverted removeStyleTags from email-renderer
no issue

Some things break in some email clients with this new setting. Disabled it for now and moved the required css style to hide the member name row to @media all.
2023-03-22 16:40:35 +01:00
Simon Backx
4eebf6612a Added image dimensions to the latest posts mobile version in email
fixes https://github.com/TryGhost/Team/issues/2799

Separate image dimensions for mobile and normal images.
2023-03-22 15:54:27 +01:00
Simon Backx
480c1a7004 Removed name from subscription details if missing
refs https://github.com/TryGhost/Team/issues/2736

If the name is not known for a member, we'll hide the name row in the subscription details in an email. This method is supported in most email clients, and requires the support of `<style>` in `<head>`.
2023-03-22 15:32:07 +01:00
Sanne de Vries
47e343ec18 Updated latest posts UI in email template
Refs https://github.com/TryGhost/Team/issues/2801
2023-03-22 15:16:55 +01:00
Simon Backx
bc0126c54e
Added subscription status text in newsletters (#16442)
fixes https://github.com/TryGhost/Team/issues/2736

Shows the actual subscription status (expires on DD MMM YYYY) in every
email when show subscription details is enabled.
2023-03-22 11:52:41 +01:00
Daniel Lockyer
247163c710
Fixed tests for verification during batch sending
refs 59df99388c

- it seems the `getSignupEvents` method is now called twice during these
  tests
2023-03-22 09:44:01 +01:00
Sanne de Vries
792bfdb498 Added UI for subscription box in newsletters
Refs https://github.com/TryGhost/Team/issues/2738

- Added preview to email newsletter settings
- Added subscription box UI to email template
2023-03-21 11:40:04 +01:00
Simon Backx
07ec33fb3a
Added latest posts to email template (#16448)
refs https://github.com/TryGhost/Team/issues/2769

Needs some extra styling and design, this is only a minimal version
behind the feature flag.
2023-03-20 14:30:42 +01:00
Peter Zimon
9d25c2a058 Fixed mobile size footer buttons in newsletters
refs. https://github.com/TryGhost/Team/issues/2740

- alignment of buttons in the footer of newsletter email template was off
2023-03-17 16:34:23 +01:00
Simon Backx
c591f1f86a Updated email batch sending snapshots 2023-03-17 16:00:14 +01:00
Sanne de Vries
48a3159d3d Added mobile feedback buttons to email template
Refs https://github.com/TryGhost/Team/issues/2740
2023-03-17 15:58:14 +01:00
Simon Backx
005b5f20fb Added E2E tests for comment CTA
fixes https://github.com/TryGhost/Team/issues/2727

Adds some E2E tests and snapshots for comment CTA in all possible situations.
2023-03-17 11:12:47 +01:00
Simon Backx
298e3da745 Fixed comment CTA button link
refs https://github.com/TryGhost/Team/issues/2672

- Use #ghost-comments instead of #ghost-comments-root
- Fixed snapshots
2023-03-17 10:27:23 +01:00
Simon Backx
423337a048 Fixed snapshots 2023-03-17 10:24:21 +01:00
Sanne de Vries
cdcb3dcd6f
Updated email template to include comment link in header and footer (#16423)
Refs https://github.com/TryGhost/Team/issues/2740

---------

Co-authored-by: Steve Larson <9larsons@gmail.com>
2023-03-16 16:10:53 +01:00
Simon Backx
450e01d1c0 Added placeholder basic subscription details to email template
refs https://github.com/TryGhost/Team/issues/2736

The renewal date is still missing, and style and desin is only a placeholder.
2023-03-15 17:12:56 +01:00
Simon Backx
80043f2699 Fixed snapshot tests in batch sending 2023-03-15 14:11:05 +01:00
Simon Backx
b27c7bc707 Fixed batch sending snapshots dates
no issue

The snapshots contained the current date, so they broke every day. This commit fixes the issue by setting a fixed date.
2023-03-15 09:45:36 +01:00
Simon Backx
bb019a0ab5 Added E2E test for sending email without post title
fixes https://github.com/TryGhost/Team/issues/2724

This change also includes new snapshots for email sending (similar for email previews, but this time for the real emails to make sure we catch changes).
2023-03-14 16:05:18 +01:00
Daniel Lockyer
89493893d1 Removed all unused variables from test files
- this cleans up all imports or variables that aren't currently being used
- this really helps keep the tests clean by only allowing what is needed
- I've left `should` as an exemption for now because we need to clean up
  how it is used
2023-03-10 14:29:55 +01:00
Simon Backx
3db434736b
🐛 Fixed replacements with fallback in plaintext newsletters (#16372)
fixes https://github.com/TryGhost/Team/issues/2683

When sending a newsletter with a replacement that has a fallback, the
replacement only happens in the HTML version of the newsletter. The
plaintext version isn't replaced.

This commit fixes the issue and adds some tests to make sure it doesn't
happen again.

The cause of the issue was that we used the original matched Regex text
to replace. But that was calculated on the HTML version, so double
quotes were encoded. This change updates the generated 'token' regex to
also match on both a double quote as the escaped double quote.
2023-03-07 15:34:43 +01:00
Simon Backx
f50224f445
Improved network access and mocking in tests (#16371)
refs https://github.com/TryGhost/Team/issues/2667

Some tests still accessed the internet. Now network access is disabled
by default. This change also introduces two helper methods related to
networking (mocking Slack and Mailgun).

This fixes two unreliable tests:
- Staff service was accessing a Slack test API -> timeout possible
- MentionSendingService was trying to send webmentions for every post
publish/change -> possible timeouts and job issues
2023-03-07 13:20:28 +01:00
Simon Backx
89ababb71f
Updated email event fetching to stop when begin and end are the same (#16326)
no issue

Optimization that makes sure we stop fetching when it is no longer
needed.
2023-02-23 15:44:01 +01:00
Simon Backx
923c522778
Implemented email analytics retrying (#16273)
fixes https://github.com/TryGhost/Team/issues/2562

New event fetching loops:
- Reworked the analytics fetching algorithm. Instead of starting again
where we stopped during the last fetching minus 30 minutes, we now just
continue where we stopped. But with ms precision (because no longer
database dependent after first fetch), and we stop at NOW - 1 minute to
reduce chance of missing events.
- Apart from that, a missing fetching loop is introduced. This fetches
events that are older than 30 minutes, and just processes all events a
second time to make sure we didn't skip any because of storage delays in
the Mailgun API.
- A new scheduled fetching loop, that allows us to schedule between a
given start/end date (currently only persisted in memory, so stops after
a reboot)

UI and endpoint changes:
- New UI to show the state of the analytics 'loops'
- New endpoint to request the analytics loop status
- New endpoint to schedule analytics
- New endpoint to cancel scheduled analytics
- Some number formatting improvements, and introduction of 'opened'
count in debug screen
- Live reload of data in the debug screen

Other changes:
- This also improves the support for maxEvents. We can now stop a
fetching loop after x events without worrying about lost events. This is
used to reduce the fetched events in the missing and scheduled event
loop (e.g. when the main one is fetching lots of events, we skip the
other loops).
- Prevents fetching the same events over and over again if no new events
come in (because we always started at the same begin timestamp). The
code increases the begin timestamp with 1 second if it is safe to do so,
to prevent the API from returning the same events over and over again.
- Some optimisations in handing the processing results (less merges to
reduce CPU usage in cases we have lots of events).

Testing:
- You can test with lots of events using the new mailgun mocking server
(Toolbox repo `scripts/mailgun-mock-server`). This can also simulate
events that are only returned after x minutes because of storage delays.
2023-02-20 16:44:13 +01:00
Simon Backx
48f9485f46
🐛 Fixed storing email failures with an empty message (#16260)
no issue

- When we receive an email failure with an empty message, the saving of
the model would fail because of schema validation that requires strings
to be non-empty.
- This adds more logging to the email analytics service to help debug
future issues
- Performance improvement to storing delivered, opened and failed emails
by replacing COALESCE with WHERE X IS NULL (tested and should give a
decent performance boost locally).
2023-02-13 15:25:36 +01:00
Simon Backx
ea2c69565f
Moved email event fetching to main thread (#16231) 2023-02-09 09:36:39 +01:00
Simon Backx
19b9696fe2 🐛 Fixed HTML escaping of feature_image_caption in newsletters
no issue

feature_image_caption was escaped in the new email stability flow, while that should not happen (bold/underline/...).
2023-01-30 14:39:08 +01:00
Simon Backx
8f8ca481a6
Fixed configUtils and adapter cache issues in E2E tests (#16167)
no issue

There are a couple of issues with resetting the Ghost instance between
E2E test files:

These issues came to the surface because of new tests written in
https://github.com/TryGhost/Ghost/pull/16117

**1. configUtils.restore does not work correctly**
`config.reset()` is a callback based method. On top of that, it doesn't
really work reliably (https://github.com/indexzero/nconf/issues/93)

What kinda happens, is that you first call `config.reset` but
immediately after you correcty reset the config using the `config.set`
calls afterwards. But since `config.reset` is async, that reset will
happen after all those sets, and the end result is that it isn't reset
correctly.

This mainly caused issues in the new updated images tests, which were
updating the config `imageOptimization.contentImageSizes`, which is a
deeply nested config value. Maybe some references to objects are reused
in nconf that cause this issue?

Wrapping `config.reset()` in a promise does fix the issue.

**2. Adapters cache not reset between tests**
At the start of each test, we set `paths:contentPath` to a nice new
temporary directory. But if a previous test already requests a
localStorage adapter, that adapter would have been created and in the
constructor `paths:contentPath` would have been passed. That same
instance will be reused in the next test run. So it won't read the new
config again. To fix this, we need to reset the adapter instances
between E2E tests.

How was this visible? Test uploads were stored in the actual git
repository, and not in a temporary directory. When writing the new image
upload tests, this also resulted in unreliable test runs because some
image names were already taken (from previous test runs).

**3. Old 2E2 test Ghost server not stopped**
Sometimes we still need access to the frontend test server using
`getAgentsWithFrontend`. But that does start a new Ghost server which is
actually listening for HTTP traffic. This could result in a fatal error
in tests because the port is already in use. The issue is that old E2E
tests also start a HTTP server, but they don't stop the server. When you
used the old `startGhost` util, it would check if a server was already
running and stop it first. The new `getAgentsWithFrontend` now also has
the same functionality to fix that issue.
2023-01-30 14:06:20 +01:00
Simon Backx
f4c55d123c
🐛 Fixed email segment generation (#16182)
fixes https://github.com/TryGhost/Team/issues/2484

The flow only send the email to segments that were targeted in the email
content. But if a part of the email is only visible for `status:free`,
that doesn't mean we don't want to send the email to `status:-free`.
This has been corrected in the new email flow.
2023-01-25 14:56:37 +01:00
Simon Backx
e879406659
Added outbound link tagging setting (#16146)
fixes https://github.com/TryGhost/Team/issues/2432
    
Adds outbound_link_tagging setting (enabled by default and behind
feature flag). If the feature flag is enabled, and the setting is
disabled, we won't add ?ref to links in emails.
    
This includes new E2E tests for email click tracking, which were also
extended to check outbound link tagging (for both MEGA and the new email
stability flow).

Also fixes a test fixture for the comments_enabled setting.
2023-01-20 13:41:36 +01:00
Simon Backx
8a2303413d Restricted email-service package to 100% test coverage
fixes https://github.com/TryGhost/Team/issues/2339

The email service is now fully covered by tests, and this commit also forces the test coverage to remain 100% after future changes.
2023-01-11 13:54:26 +01:00
Simon Backx
117afb232c Improved stability of batch sending test
refs https://github.com/TryGhost/Ghost/actions/runs/3884901582/jobs/6628075997

The 'Doesn't include members created after the email in the batches' test sometimes failed randomly because of timing. This change ensures a strict timing.
2023-01-10 17:12:56 +01:00
Simon Backx
9ca2e3f183 🐛 Fixed storing email recipient failures
fixes https://github.com/TryGhost/Team/issues/2398

There was an error when fetching the existing email recipient failure. It ended up matching all recipient failures. The result was that only one failure was stored in the database.
2023-01-10 15:41:42 +01:00
Fabien "egg" O'Carroll
953f3856a8 Handled EmailBounceEvent with 605 error code
When Mailgun fails to deliver an email to an address because the
address has already bounced before, it gives us a permanent fail event
with a 605 error code rather than a 5xx one. Because we want to
"backfill" our suppressions data with previously bounced email
addresses, we want to handle this specific error code.

We may update this logic in the future based on new information from
Mailgun with respect to their 6xx error codes and the
meanings/underlying cause of theme.

This also moves the tests which check for whether or not emails are
suppressed into their own fail so that we do not pollute the event
storage tests, and adds more tests cases.

We also fix a leaky sinon stub which we were not resetting in the email
event storage tests
2023-01-05 17:11:37 +07:00
Simon Backx
913ad18b71
Added DomainEvents.allSettled utility method (#16075)
no issue

With the increased usage of DomainEvents, it gets harder to build
reliable tests without having to resort to timeouts. This utility method
allows us to wait for all events to be processed before continuing with
the test.

This change should speed up tests and make them more reliable.

It only adds extra code when running tests and shouldn't impact
production.
2023-01-04 14:30:35 +01:00
Simon Backx
819d0d884c
Improved email verification required checks (#16060)
fixes https://github.com/TryGhost/Team/issues/2366
refs https://ghost.slack.com/archives/C02G9E68C/p1670232405014209

Probem described in issue.

In the old MEGA flow:
- The `email_verification_required` check is now repeated inside the job

In the new email service flow:
- The `email_verification_required` is now checked (didn't happen
before)
- When generating the email batch recipients, we only include members
that were created before the email was created. That way it is
impossible to avoid limit checks by inserting new members between
creating an email and sending an email.
- We don't need to repeat the check inside the job because of the above
changes

Improved handling of large imports:
- When checking `email_verification_required`, we now also check if the
import threshold is reached (a new method is introduced in
vertificationTrigger specifically for this usage). If it is, we start
the verification progress. This is required for long running imports
that only check the verification threshold at the very end.
- This change increases the concurrency of fastq to 3 (refs
https://ghost.slack.com/archives/C02G9E68C/p1670232405014209). So when
running a long import, it is now possible to send emails without having
to wait for the import. Above change makes sure it is not possible to
get around the verification limits.

Refactoring:
- Removed the need to use `updateVerificationTrigger` by making
thresholds getters instead of fixed variables.
- Improved awaiting of members import job in regression test
2023-01-04 11:22:12 +01:00
Fabien 'egg' O'Carroll
e78612bb66
Fixed MailgunEmailSuppressionList adding non-5xx failures to the list
The MailgunEmailSuppression list was incorrectly adding emails
to the suppression list for permanent failure events which have
an error code outside of the 5xx range.
2023-01-04 17:03:52 +07:00
Simon Backx
47cd7a7095
🐛 Handled unknown Mailgun events (#15995)
refs https://ghost.slack.com/archives/C02G9E68C/p1670916538764019

- We receive events that don't have an emailId or providerId.
- We filter those events now and log them as an error
2022-12-14 11:17:45 +01:00
Fabien "egg" O'Carroll
939e3ce96f Fixed suppression list handling of spam complaint events
refs https://github.com/TryGhost/Team/issues/2351

We were storing all suppressions with a reason of bounced, rather than
spam for spam complaint events.
2022-12-08 13:02:36 +07:00
Fabien "egg" O'Carroll
5749a17910 Cleaned up EmailEventStorage tests
This ensures that services aren't required before boot, so that they are
initialised in the correct order.
2022-12-08 13:02:36 +07:00
Fabien "egg" O'Carroll
9736d942e1 Unsubscribed Members from newsletters when their email is suppressed
refs https://github.com/TryGhost/Team/issues/2367

This ensures that a Member is not considered subscribed to any emails, so that
counts for newsletter recipients are correct. Eventually we will filter members
on their email suppression status but this is not implemented yet.
2022-12-08 13:02:36 +07:00
Ghost CI
54b693a359 Merged v5.25.1 into main 2022-12-06 05:14:41 +00:00
Simon Backx
c47891c3f6
🐛 Fixed setting delivered_at to null after hard bounce (#15942)
refs https://ghost.slack.com/archives/C02G9E68C/p1670075366333929?thread_ts=1669963540.980309&cid=C02G9E68C

When we receive a permanent bounce/failure, we set delivered_at to null.
But we don't want to lose this information.

Instead we should be able to handle recipients that both have failed_at
and delivered_at set.
2022-12-06 10:26:54 +05:30