Commit Graph

44 Commits

Author SHA1 Message Date
Simon Backx
b7361e2f1f Fixed email service test coverage for limitImageWidth
refs 74f25faf61

Changes weren't covered by unit tests.
2023-03-22 14:17:01 +01:00
Simon Backx
bc0126c54e
Added subscription status text in newsletters (#16442)
fixes https://github.com/TryGhost/Team/issues/2736

Shows the actual subscription status (expires on DD MMM YYYY) in every
email when show subscription details is enabled.
2023-03-22 11:52:41 +01:00
Simon Backx
07ec33fb3a
Added latest posts to email template (#16448)
refs https://github.com/TryGhost/Team/issues/2769

Needs some extra styling and design, this is only a minimal version
behind the feature flag.
2023-03-20 14:30:42 +01:00
Simon Backx
92fc5bdf70 Fixed email service unit tests for duplicate email buttons 2023-03-17 16:09:50 +01:00
Simon Backx
25f4974846 Added comment CTA in newsletters
fixes https://github.com/TryGhost/Team/issues/2672

- Moves the feature from behind the flag
- Also hides the comment CTA for email only posts

Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
2023-03-17 09:38:55 +01:00
Simon Backx
450e01d1c0 Added placeholder basic subscription details to email template
refs https://github.com/TryGhost/Team/issues/2736

The renewal date is still missing, and style and desin is only a placeholder.
2023-03-15 17:12:56 +01:00
Simon Backx
66c20710ba Added placeholder comment CTA button to emails
fixes https://github.com/TryGhost/Team/issues/2725

- Added to emails if labs flag enabled, comments enabled and comment CTA button enabled
- Links to comment section
- Design and styling not added yet
2023-03-14 17:11:24 +01:00
Simon Backx
1e435fb328
Implemented showPostTitleSection (#16404)
fixes https://github.com/TryGhost/Team/issues/2705

- Added showPostTitleSection to newsletter model in admin
- Wired up UI to admin model so it saves to the database
- Implemented showPostTitleSection in newsletter preview and added some
minor temporary css styling
- Implemented showPostTitleSection in newsletter template in backend,
and added some extra CSS styling to fix spacing
2023-03-14 11:29:43 +01:00
Simon Backx
832610fd2a
🐛 Fixed retrying failed emails when rescheduling them (#16383)
fixes https://github.com/TryGhost/Team/issues/2560

When an email fails, and you reschedule the post, the error dialog was
shown (from the previous try). The retry button on that page allowed you
to retry sending the email immediately, which could be very confusing.

- The email error dialog is no longer shown for scheduled emails
- The email status is no longer polled for scheduled emails
- Retrying an email is not possible via the API if the post status is
not published or sent
- Added some extra snapshot tests
- When retrying an email, we immediately update the email status to
'pending' to have a better API response (instead of still returning
failed).
- Disabled email sending retrying in development (otherwise very hard to
test failed emails if it takes 10 mins before it gives up automatic
retrying)
2023-03-09 12:32:22 +01:00
Simon Backx
3db434736b
🐛 Fixed replacements with fallback in plaintext newsletters (#16372)
fixes https://github.com/TryGhost/Team/issues/2683

When sending a newsletter with a replacement that has a fallback, the
replacement only happens in the HTML version of the newsletter. The
plaintext version isn't replaced.

This commit fixes the issue and adds some tests to make sure it doesn't
happen again.

The cause of the issue was that we used the original matched Regex text
to replace. But that was calculated on the HTML version, so double
quotes were encoded. This change updates the generated 'token' regex to
also match on both a double quote as the escaped double quote.
2023-03-07 15:34:43 +01:00
Simon Backx
7781c2da07 Implemented retrying for sending email via Mailgun API
no issue

Retry sending an email up to 5 times if it failed.
2023-02-21 16:06:23 +01:00
Simon Backx
8dfa490638
Added outbound link tagging for web posts (#16201)
fixes https://github.com/TryGhost/Team/issues/2433

- Moved all outbound link tagging code to separate OutboundLinkTagger
- Because a site can easily enable/disable this feature, we don't store
the ?refs in the HTML but add them on the fly for now in the Content
API.
2023-02-16 11:26:35 +01:00
Simon Backx
0e8d13bdde 🐛 Fixed email replacements without fallback
fixes https://github.com/TryGhost/Team/issues/2557

When a member doen't have a name, and the first_name replacement doesn't have a fallback, we did show %recipient.first_name% instead of an empty string.
2023-02-13 15:58:40 +01:00
Simon Backx
48f9485f46
🐛 Fixed storing email failures with an empty message (#16260)
no issue

- When we receive an email failure with an empty message, the saving of
the model would fail because of schema validation that requires strings
to be non-empty.
- This adds more logging to the email analytics service to help debug
future issues
- Performance improvement to storing delivered, opened and failed emails
by replacing COALESCE with WHERE X IS NULL (tested and should give a
decent performance boost locally).
2023-02-13 15:25:36 +01:00
Simon Backx
ea2c69565f
Moved email event fetching to main thread (#16231) 2023-02-09 09:36:39 +01:00
Simon Backx
fb3cbe5fc8
Added short lived caching to email batch body (#16233)
fixes https://github.com/TryGhost/Team/issues/2522

When sending an email for multiple batches at the same time, we now
reuse the same email body for each batch in the same segment. This
reduces the amount of database queries and makes the sending more
reliable in case of database failures.

The cache is short lived. After sending the email it is automatically
garbage collected.
2023-02-07 11:01:49 +01:00
Simon Backx
0f08f39a1b 🐛 Fixed link to posts on audio cards in emails
refs https://ghost.slack.com/archives/CTH5NDJMS/p1675356984043299

The url property doesn't exists on post, this was copied from the old MEGA flow that serialized the model first.
2023-02-02 18:19:31 +01:00
Simon Backx
06de10786f
Added automatic database query retrying to the email service (#16218)
fixes https://github.com/TryGhost/Team/issues/2512

The email sending is a crucial flow that should not be interrupted. If
there are database connection issues, we should try to recover from them
automatically by retrying individual database queries and transactions
for a limited amount of time.

This adds a helper method to retry database operations. It limits all
database queries before sending the actual email to maximum 10 minutes.
Database operations that happen after sending have a higher retry time
because they are more crucial to prevent data loss (e.g. saving that an
email was sent).
2023-02-02 14:12:54 +01:00
Simon Backx
c798f383f9 🐛 Fixed not reactivating email analytics jobs
no issue

When a site doesn't have any emails on boot, it doesn't schedule the email analytics job. With this change, the new email flow will also restart that job after an email has been created.
2023-01-26 17:35:45 +01:00
Simon Backx
5e1e6c9863
Improved email error logging (#16184)
no issue

Logs errors to Sentry and adds error codes.
2023-01-25 14:57:10 +01:00
Simon Backx
f4c55d123c
🐛 Fixed email segment generation (#16182)
fixes https://github.com/TryGhost/Team/issues/2484

The flow only send the email to segments that were targeted in the email
content. But if a part of the email is only visible for `status:free`,
that doesn't mean we don't want to send the email to `status:-free`.
This has been corrected in the new email flow.
2023-01-25 14:56:37 +01:00
Simon Backx
4b0ca9399d 🐛 Reduced concurrency when fetching Mailgun events (#16176)
refs https://github.com/TryGhost/Team/issues/2482

This change adds a small sleep in between dispatching events in the
worker thread that reads the events from Mailgun. That should reduce the
amount of queries we fire parallel to each other and could cause the
connection pool to run out of connections.

It also reduces the amount of concurrent sending to 2 from 10. Also to
make sure the connection pool doesn't run out of connections while
sending emails, and to reduce the chance of new connections falling back
on a (delayed) replicated database.
2023-01-25 13:07:01 +01:00
Rishabh Garg
88979c852b
Updated email sending to remove invalid recipient emails (#16171)
closes https://github.com/TryGhost/Team/issues/2388

We have seen examples of sites with member emails that have invalid characters that can cause an entire email send to fail, or just cause a failure to those addresses. The issue that allowed members with invalid email address to be saved was patched earlier, but its possible there are still sites that contain some of those invalid email addresses.

This change updates new sending service to filter out the recipients with invalid email address before passing them to mail provider, so these rogue addresses don't affect the whole batch in anyway. We also trim the recipient emails to clear out any spaces first, which is the most likely culprit.

- uses new email validator that detects invalid email addresses with special chars
2023-01-24 16:13:10 +05:30
Simon Backx
3ac282598c Fixed Playwright tests for new email flow
no issue

Tests stopped working because the Mailgun mocker stopped working since we moved to the new email flow.

This also fixes a unit test that needed to get updated.
2023-01-20 19:00:13 +01:00
Simon Backx
e879406659
Added outbound link tagging setting (#16146)
fixes https://github.com/TryGhost/Team/issues/2432
    
Adds outbound_link_tagging setting (enabled by default and behind
feature flag). If the feature flag is enabled, and the setting is
disabled, we won't add ?ref to links in emails.
    
This includes new E2E tests for email click tracking, which were also
extended to check outbound link tagging (for both MEGA and the new email
stability flow).

Also fixes a test fixture for the comments_enabled setting.
2023-01-20 13:41:36 +01:00
Simon Backx
8a2303413d Restricted email-service package to 100% test coverage
fixes https://github.com/TryGhost/Team/issues/2339

The email service is now fully covered by tests, and this commit also forces the test coverage to remain 100% after future changes.
2023-01-11 13:54:26 +01:00
Simon Backx
ee36a0fce9 Added 100% test coverage to SendingService
refs https://github.com/TryGhost/Team/issues/2339
2023-01-11 13:39:25 +01:00
Simon Backx
6a664d11b9 Added 100% test coverage to email renderer and fixed authors bug
refs https://github.com/TryGhost/Team/issues/2339

This fixed a bug in the new email flow that more than 2 authors were displayed as 'undefined & 2 others'.
2023-01-11 12:14:02 +01:00
Simon Backx
6a364e7779 Added 100% test coverage for EmailEventStorage
refs https://github.com/TryGhost/Team/issues/2339
2023-01-10 16:36:41 +01:00
Simon Backx
bdf45fad84 Added 100% test coverage to BatchSendingService
refs https://github.com/TryGhost/Team/issues/2339
2023-01-10 14:58:50 +01:00
Elena Baidakova
69d4a96fb8
🎨 Add ability to send test email with chosen newsletter (#15783)
closes TryGhost/Team#1897

-
[Design](https://www.figma.com/file/RpEbPA7H7VHLtXjt3YyB2t/Multiple-Newsletters-Preview?node-id=0%3A1&t=YkDXy063OkCrAI4a-0)
-
[Discussion](https://ghost.slack.com/archives/C019B1K4FAM/p1667924062808939)
2023-01-09 17:48:30 +04:00
Simon Backx
30853bc213 Added 100% test coverage to EmailEventProcessor
refs https://github.com/TryGhost/Team/issues/2339
2023-01-04 16:07:19 +01:00
Simon Backx
c31bc58210 Added 100% test coverage to EmailController
refs https://github.com/TryGhost/Team/issues/2339
2023-01-04 15:22:49 +01:00
Simon Backx
e2e9a56583 Added 100% test coverage to EmailService class
refs https://github.com/TryGhost/Team/issues/2339

Tests the EmailService class of the email-service package.
2023-01-04 14:25:29 +01:00
Rishabh Garg
a09e86da05
Fixed email preview text with new stability flow (#15996)
closes https://github.com/TryGhost/Team/issues/2382

The preview text is getting set to subject line in the new email flow so it repeats multiple times in the inbox(subject+preview+title). This was because the new flow doesn't use the post serialisation that the old system did, causing excerpt to be empty in the email rendering.

Old system was using post serialisation here -
a721e4f2d7/ghost/core/core/server/services/mega/post-email-serializer.js (L136-L139).

This change adds explicit method to calculate the preview text for email in email renderer service using same logic as used in old system.

Co-authored-by: Simon Backx <git@simonbackx.com>
2022-12-14 15:54:26 +05:30
Simon Backx
9e6f5e93d8
🐛 Fixed email header images serving original image size (#15950)
fixes https://github.com/TryGhost/Team/issues/2368

- Removed the usage of the `isLocalContentImage` Koenig util for the
email header and feature image url generation.
- While we were trying to set the width to 1200px, we didn't have that
size hardcoded. So that url would redirect back to the original location
instead of serving a smaller image. So I added a new internal size to
the `imageOptimization` config.
- This is fixed in both the new and old email flow and includes some
extra tests for the new flow.
2022-12-09 11:17:22 +01:00
Simon Backx
c47891c3f6
🐛 Fixed setting delivered_at to null after hard bounce (#15942)
refs https://ghost.slack.com/archives/C02G9E68C/p1670075366333929?thread_ts=1669963540.980309&cid=C02G9E68C

When we receive a permanent bounce/failure, we set delivered_at to null.
But we don't want to lose this information.

Instead we should be able to handle recipients that both have failed_at
and delivered_at set.
2022-12-06 10:26:54 +05:30
Simon Backx
1958d0db63
Removed spaces and quotes from replacement ids (#15921)
fixes https://github.com/TryGhost/Team/issues/2350
2022-12-02 10:49:01 +01:00
Simon Backx
6fbda0730d Added unit test for paywall and segments 2022-12-02 10:48:29 +01:00
Simon Backx
53b5c152d9 Added email event storage unit tests
refs https://github.com/TryGhost/Team/issues/2339
2022-12-02 10:21:39 +01:00
Rishabh Garg
f9161876b3
Updated unit test coverage for email service (#15913)
refs https://github.com/TryGhost/Team/issues/2339

- adds unit test for email renderer, segmenter and sending service
2022-12-02 00:33:28 +05:30
Rishabh Garg
42f9d392a3
Added mailgun provider for sending emails (#15896)
closes https://github.com/TryGhost/Team/issues/2309

- adds new mailgun provider to send out batch emails
- updates sending service to send email id for mailgun provider, allows tagging mail with email id
2022-11-30 16:21:58 +05:30
Simon Backx
f5045b9bf7
Added email renderer implementation draft (#15877)
fixes https://github.com/TryGhost/Team/issues/2308

- Still has some missing pieces, but mostly works.
- Uses new handlebars template for emails
- When sending emails with the new email stability flag enabled, one
test email is now sent via the default smtp ghost mailer.
2022-11-29 11:27:17 +01:00
Simon Backx
44f189b56a
Added email service package (#15849)
fixes https://github.com/TryGhost/Team/issues/2282

Added a new email service package that is used when the email stability
flag is enabled. Currently not yet implemented so will throw an error
for all entry points (if flag enabled).

Removed usage of `labs.isSet.bind` across the code, because that breaks
the stubbing of labs by `mockManager.mockLabsEnabled` and
`mockManager.mockLabsDisabled`. `flag => labs.isSet(flag)` should be
used instead.

All email depending tests now disable the `emailStability` feature flag
to keep the tests passing + make sure we still run all the tests for the
old flow while the email stability package is being built.
2022-11-21 10:29:53 +01:00