fixes https://github.com/TryGhost/Team/issues/2522
When sending an email for multiple batches at the same time, we now
reuse the same email body for each batch in the same segment. This
reduces the amount of database queries and makes the sending more
reliable in case of database failures.
The cache is short lived. After sending the email it is automatically
garbage collected.
fixes https://github.com/TryGhost/Team/issues/2512
The email sending is a crucial flow that should not be interrupted. If
there are database connection issues, we should try to recover from them
automatically by retrying individual database queries and transactions
for a limited amount of time.
This adds a helper method to retry database operations. It limits all
database queries before sending the actual email to maximum 10 minutes.
Database operations that happen after sending have a higher retry time
because they are more crucial to prevent data loss (e.g. saving that an
email was sent).
no issue
When a site doesn't have any emails on boot, it doesn't schedule the email analytics job. With this change, the new email flow will also restart that job after an email has been created.
fixes https://github.com/TryGhost/Team/issues/2484
The flow only send the email to segments that were targeted in the email
content. But if a part of the email is only visible for `status:free`,
that doesn't mean we don't want to send the email to `status:-free`.
This has been corrected in the new email flow.
refs https://github.com/TryGhost/Team/issues/2482
This change adds a small sleep in between dispatching events in the
worker thread that reads the events from Mailgun. That should reduce the
amount of queries we fire parallel to each other and could cause the
connection pool to run out of connections.
It also reduces the amount of concurrent sending to 2 from 10. Also to
make sure the connection pool doesn't run out of connections while
sending emails, and to reduce the chance of new connections falling back
on a (delayed) replicated database.
closes https://github.com/TryGhost/Team/issues/2388
We have seen examples of sites with member emails that have invalid characters that can cause an entire email send to fail, or just cause a failure to those addresses. The issue that allowed members with invalid email address to be saved was patched earlier, but its possible there are still sites that contain some of those invalid email addresses.
This change updates new sending service to filter out the recipients with invalid email address before passing them to mail provider, so these rogue addresses don't affect the whole batch in anyway. We also trim the recipient emails to clear out any spaces first, which is the most likely culprit.
- uses new email validator that detects invalid email addresses with special chars
no issue
Tests stopped working because the Mailgun mocker stopped working since we moved to the new email flow.
This also fixes a unit test that needed to get updated.
fixes https://github.com/TryGhost/Team/issues/2432
Adds outbound_link_tagging setting (enabled by default and behind
feature flag). If the feature flag is enabled, and the setting is
disabled, we won't add ?ref to links in emails.
This includes new E2E tests for email click tracking, which were also
extended to check outbound link tagging (for both MEGA and the new email
stability flow).
Also fixes a test fixture for the comments_enabled setting.
refs https://github.com/TryGhost/Toolbox/issues/501
- this reverts commit 48dda23554
- also includes a resolution for `@elastic/elasticsearch` so we don't
run a version that is potentially problematic - see referenced issue
for context
fixes https://github.com/TryGhost/Team/issues/2383
A user could use `{uuid}` inside an email only content and it would work. This currently isn't supposed to be used outside internal features (link click tracking, feedback buttons). For now this is only fixed in the new email flow under the email stability flag.
fixes https://github.com/TryGhost/Team/issues/2339
The email service is now fully covered by tests, and this commit also forces the test coverage to remain 100% after future changes.
fixes https://github.com/TryGhost/Team/issues/2398
There was an error when fetching the existing email recipient failure. It ended up matching all recipient failures. The result was that only one failure was stored in the database.
fixes https://github.com/TryGhost/Team/issues/2366
refs https://ghost.slack.com/archives/C02G9E68C/p1670232405014209
Probem described in issue.
In the old MEGA flow:
- The `email_verification_required` check is now repeated inside the job
In the new email service flow:
- The `email_verification_required` is now checked (didn't happen
before)
- When generating the email batch recipients, we only include members
that were created before the email was created. That way it is
impossible to avoid limit checks by inserting new members between
creating an email and sending an email.
- We don't need to repeat the check inside the job because of the above
changes
Improved handling of large imports:
- When checking `email_verification_required`, we now also check if the
import threshold is reached (a new method is introduced in
vertificationTrigger specifically for this usage). If it is, we start
the verification progress. This is required for long running imports
that only check the verification threshold at the very end.
- This change increases the concurrency of fastq to 3 (refs
https://ghost.slack.com/archives/C02G9E68C/p1670232405014209). So when
running a long import, it is now possible to send emails without having
to wait for the import. Above change makes sure it is not possible to
get around the verification limits.
Refactoring:
- Removed the need to use `updateVerificationTrigger` by making
thresholds getters instead of fixed variables.
- Improved awaiting of members import job in regression test
- this was all getting terribly behind so I've done several things:
- majority of `@tryghost/*` except Lexical packages
- gscan + knex-migrator to remove old `@tryghost/errors` usage
- bumped lockfile
closes https://github.com/TryGhost/Team/issues/2382
The preview text is getting set to subject line in the new email flow so it repeats multiple times in the inbox(subject+preview+title). This was because the new flow doesn't use the post serialisation that the old system did, causing excerpt to be empty in the email rendering.
Old system was using post serialisation here -
a721e4f2d7/ghost/core/core/server/services/mega/post-email-serializer.js (L136-L139).
This change adds explicit method to calculate the preview text for email in email renderer service using same logic as used in old system.
Co-authored-by: Simon Backx <git@simonbackx.com>
fixes https://github.com/TryGhost/Team/issues/2368
- Removed the usage of the `isLocalContentImage` Koenig util for the
email header and feature image url generation.
- While we were trying to set the width to 1200px, we didn't have that
size hardcoded. So that url would redirect back to the original location
instead of serving a smaller image. So I added a new internal size to
the `imageOptimization` config.
- This is fixed in both the new and old email flow and includes some
extra tests for the new flow.
refs https://ghost.slack.com/archives/C02G9E68C/p1670215917451249
When a member is deleted, and we receive an opened event for an email to
that member. We threw an uncaught Bookshelf EmptyResponse error.
- This change makes fetching the member not a requirement when handling
that event in the last seen at updater.
- It also adds try catches for all event listeners in the last seen at
updater
refs https://github.com/TryGhost/Team/issues/2339
- Includes a new pattern in the job manager that allows us to properly
await jobs.
- Added new convenience mocking methods to stub settings
- Tests the main flows for bulk sending:
- Sending in multiple batches
- Sending to multiple segments
- Handling a failed batch and retrying that batch
- Fixes bug in batch generation (ordering not working)
In a different PR I'll add more detailed tests.
