Commit Graph

59 Commits

Author SHA1 Message Date
Simon Backx
66c20710ba Added placeholder comment CTA button to emails
fixes https://github.com/TryGhost/Team/issues/2725

- Added to emails if labs flag enabled, comments enabled and comment CTA button enabled
- Links to comment section
- Design and styling not added yet
2023-03-14 17:11:24 +01:00
Simon Backx
1e435fb328
Implemented showPostTitleSection (#16404)
fixes https://github.com/TryGhost/Team/issues/2705

- Added showPostTitleSection to newsletter model in admin
- Wired up UI to admin model so it saves to the database
- Implemented showPostTitleSection in newsletter preview and added some
minor temporary css styling
- Implemented showPostTitleSection in newsletter template in backend,
and added some extra CSS styling to fix spacing
2023-03-14 11:29:43 +01:00
Simon Backx
832610fd2a
🐛 Fixed retrying failed emails when rescheduling them (#16383)
fixes https://github.com/TryGhost/Team/issues/2560

When an email fails, and you reschedule the post, the error dialog was
shown (from the previous try). The retry button on that page allowed you
to retry sending the email immediately, which could be very confusing.

- The email error dialog is no longer shown for scheduled emails
- The email status is no longer polled for scheduled emails
- Retrying an email is not possible via the API if the post status is
not published or sent
- Added some extra snapshot tests
- When retrying an email, we immediately update the email status to
'pending' to have a better API response (instead of still returning
failed).
- Disabled email sending retrying in development (otherwise very hard to
test failed emails if it takes 10 mins before it gives up automatic
retrying)
2023-03-09 12:32:22 +01:00
Simon Backx
3db434736b
🐛 Fixed replacements with fallback in plaintext newsletters (#16372)
fixes https://github.com/TryGhost/Team/issues/2683

When sending a newsletter with a replacement that has a fallback, the
replacement only happens in the HTML version of the newsletter. The
plaintext version isn't replaced.

This commit fixes the issue and adds some tests to make sure it doesn't
happen again.

The cause of the issue was that we used the original matched Regex text
to replace. But that was calculated on the HTML version, so double
quotes were encoded. This change updates the generated 'token' regex to
also match on both a double quote as the escaped double quote.
2023-03-07 15:34:43 +01:00
Simon Backx
a58396b2de Removed email sending retrying in test mode
no issue

Disables email sending retrying in test mode by default. This is to prevent test timeouts and to make testing more reliable in case where we manually let a batch fail.
2023-03-07 11:52:10 +01:00
Daniel Lockyer
42768937fd Disabled autoRefresh when adding an email recipient failure
- we don't need the updated model once we've saved it in the DB, so we
  can disable the auto refresh in Bookshelf to save a query
2023-02-27 15:45:25 +01:00
Simon Backx
fb210ba6a0 Added extra Mailgun retry
no issue
2023-02-21 16:22:26 +01:00
Simon Backx
7781c2da07 Implemented retrying for sending email via Mailgun API
no issue

Retry sending an email up to 5 times if it failed.
2023-02-21 16:06:23 +01:00
Simon Backx
b665b1a3cc Implemented email analytics retrying (#16273)
fixes https://github.com/TryGhost/Team/issues/2562

New event fetching loops:
- Reworked the analytics fetching algorithm. Instead of starting again
where we stopped during the last fetching minus 30 minutes, we now just
continue where we stopped. But with ms precision (because no longer
database dependent after first fetch), and we stop at NOW - 1 minute to
reduce chance of missing events.
- Apart from that, a missing fetching loop is introduced. This fetches
events that are older than 30 minutes, and just processes all events a
second time to make sure we didn't skip any because of storage delays in
the Mailgun API.
- A new scheduled fetching loop, that allows us to schedule between a
given start/end date (currently only persisted in memory, so stops after
a reboot)

UI and endpoint changes:
- New UI to show the state of the analytics 'loops'
- New endpoint to request the analytics loop status
- New endpoint to schedule analytics
- New endpoint to cancel scheduled analytics
- Some number formatting improvements, and introduction of 'opened'
count in debug screen
- Live reload of data in the debug screen

Other changes:
- This also improves the support for maxEvents. We can now stop a
fetching loop after x events without worrying about lost events. This is
used to reduce the fetched events in the missing and scheduled event
loop (e.g. when the main one is fetching lots of events, we skip the
other loops).
- Prevents fetching the same events over and over again if no new events
come in (because we always started at the same begin timestamp). The
code increases the begin timestamp with 1 second if it is safe to do so,
to prevent the API from returning the same events over and over again.
- Some optimisations in handing the processing results (less merges to
reduce CPU usage in cases we have lots of events).

Testing:
- You can test with lots of events using the new mailgun mocking server
(Toolbox repo `scripts/mailgun-mock-server`). This can also simulate
events that are only returned after x minutes because of storage delays.
2023-02-21 16:00:32 +01:00
Simon Backx
8dfa490638
Added outbound link tagging for web posts (#16201)
fixes https://github.com/TryGhost/Team/issues/2433

- Moved all outbound link tagging code to separate OutboundLinkTagger
- Because a site can easily enable/disable this feature, we don't store
the ?refs in the HTML but add them on the fly for now in the Content
API.
2023-02-16 11:26:35 +01:00
Peter Zimon
5bb5977a47 Fixed issue with bookmark cards width on Android
no refs.

- On Android Gmail client the bookmark cards were not taking the full width of the screen
2023-02-14 14:31:16 +01:00
Simon Backx
0e8d13bdde 🐛 Fixed email replacements without fallback
fixes https://github.com/TryGhost/Team/issues/2557

When a member doen't have a name, and the first_name replacement doesn't have a fallback, we did show %recipient.first_name% instead of an empty string.
2023-02-13 15:58:40 +01:00
Simon Backx
48f9485f46
🐛 Fixed storing email failures with an empty message (#16260)
no issue

- When we receive an email failure with an empty message, the saving of
the model would fail because of schema validation that requires strings
to be non-empty.
- This adds more logging to the email analytics service to help debug
future issues
- Performance improvement to storing delivered, opened and failed emails
by replacing COALESCE with WHERE X IS NULL (tested and should give a
decent performance boost locally).
2023-02-13 15:25:36 +01:00
Simon Backx
ea2c69565f
Moved email event fetching to main thread (#16231) 2023-02-09 09:36:39 +01:00
Simon Backx
fb3cbe5fc8
Added short lived caching to email batch body (#16233)
fixes https://github.com/TryGhost/Team/issues/2522

When sending an email for multiple batches at the same time, we now
reuse the same email body for each batch in the same segment. This
reduces the amount of database queries and makes the sending more
reliable in case of database failures.

The cache is short lived. After sending the email it is automatically
garbage collected.
2023-02-07 11:01:49 +01:00
Simon Backx
30b9d02f70 Increased wait time between Mailgun events to 200ms
no issue

Reduce concurrency if the database is going a bit slower, until we have more permanent fix for this.
2023-02-03 14:38:44 +01:00
Simon Backx
29e1e93ca1 Increased wait time between Mailgun events to 150ms
no issue

Reduce concurrency if the database is going a bit slower, until we have more permanent fix for this.
2023-02-03 10:39:55 +01:00
Simon Backx
4fad1986a0 🐛 Fixed link to posts on audio cards in emails
refs https://ghost.slack.com/archives/CTH5NDJMS/p1675356984043299

The url property doesn't exists on post, this was copied from the old MEGA flow that serialized the model first.
2023-02-03 10:37:02 +01:00
Simon Backx
06de10786f
Added automatic database query retrying to the email service (#16218)
fixes https://github.com/TryGhost/Team/issues/2512

The email sending is a crucial flow that should not be interrupted. If
there are database connection issues, we should try to recover from them
automatically by retrying individual database queries and transactions
for a limited amount of time.

This adds a helper method to retry database operations. It limits all
database queries before sending the actual email to maximum 10 minutes.
Database operations that happen after sending have a higher retry time
because they are more crucial to prevent data loss (e.g. saving that an
email was sent).
2023-02-02 14:12:54 +01:00
Simon Backx
b50e3915ea 🐛 Fixed HTML escaping of feature_image_caption in newsletters
no issue

feature_image_caption was escaped in the new email stability flow, while that should not happen (bold/underline/...).
2023-01-30 16:58:59 +01:00
Simon Backx
c798f383f9 🐛 Fixed not reactivating email analytics jobs
no issue

When a site doesn't have any emails on boot, it doesn't schedule the email analytics job. With this change, the new email flow will also restart that job after an email has been created.
2023-01-26 17:35:45 +01:00
Simon Backx
5e1e6c9863
Improved email error logging (#16184)
no issue

Logs errors to Sentry and adds error codes.
2023-01-25 14:57:10 +01:00
Simon Backx
f4c55d123c
🐛 Fixed email segment generation (#16182)
fixes https://github.com/TryGhost/Team/issues/2484

The flow only send the email to segments that were targeted in the email
content. But if a part of the email is only visible for `status:free`,
that doesn't mean we don't want to send the email to `status:-free`.
This has been corrected in the new email flow.
2023-01-25 14:56:37 +01:00
Simon Backx
4b0ca9399d 🐛 Reduced concurrency when fetching Mailgun events (#16176)
refs https://github.com/TryGhost/Team/issues/2482

This change adds a small sleep in between dispatching events in the
worker thread that reads the events from Mailgun. That should reduce the
amount of queries we fire parallel to each other and could cause the
connection pool to run out of connections.

It also reduces the amount of concurrent sending to 2 from 10. Also to
make sure the connection pool doesn't run out of connections while
sending emails, and to reduce the chance of new connections falling back
on a (delayed) replicated database.
2023-01-25 13:07:01 +01:00
Rishabh Garg
88979c852b
Updated email sending to remove invalid recipient emails (#16171)
closes https://github.com/TryGhost/Team/issues/2388

We have seen examples of sites with member emails that have invalid characters that can cause an entire email send to fail, or just cause a failure to those addresses. The issue that allowed members with invalid email address to be saved was patched earlier, but its possible there are still sites that contain some of those invalid email addresses.

This change updates new sending service to filter out the recipients with invalid email address before passing them to mail provider, so these rogue addresses don't affect the whole batch in anyway. We also trim the recipient emails to clear out any spaces first, which is the most likely culprit.

- uses new email validator that detects invalid email addresses with special chars
2023-01-24 16:13:10 +05:30
Sanne de Vries
4b61f23bd0 Fixed minor email preview visual bugs
No ref
2023-01-23 10:46:30 +01:00
Simon Backx
693216c29e Improved failed email error messages
refs https://ghost.slack.com/archives/C02G9E68C/p1673570945616369?thread_ts=1672700158.659209&cid=C02G9E68C

Fixed a typo in the error messages and made the errors more clear. Also hide the email configuration part when there is no email configuration.
2023-01-20 18:37:43 +01:00
Simon Backx
e879406659
Added outbound link tagging setting (#16146)
fixes https://github.com/TryGhost/Team/issues/2432
    
Adds outbound_link_tagging setting (enabled by default and behind
feature flag). If the feature flag is enabled, and the setting is
disabled, we won't add ?ref to links in emails.
    
This includes new E2E tests for email click tracking, which were also
extended to check outbound link tagging (for both MEGA and the new email
stability flow).

Also fixes a test fixture for the comments_enabled setting.
2023-01-20 13:41:36 +01:00
Simon Backx
d5a359f2db Removed possibility to use {uuid} replacements in email
fixes https://github.com/TryGhost/Team/issues/2383

A user could use `{uuid}` inside an email only content and it would work. This currently isn't supposed to be used outside internal features (link click tracking, feedback buttons). For now this is only fixed in the new email flow under the email stability flag.
2023-01-11 15:56:15 +01:00
Simon Backx
8a2303413d Restricted email-service package to 100% test coverage
fixes https://github.com/TryGhost/Team/issues/2339

The email service is now fully covered by tests, and this commit also forces the test coverage to remain 100% after future changes.
2023-01-11 13:54:26 +01:00
Simon Backx
9650521376 Fixed variable date formatting in new email flow
refs https://github.com/TryGhost/Ghost/actions/runs/3892250013/jobs/6643448457

Depending on the environment the date formatting in a newsletter could change. Now it is always set to the DD MMM YYYY format (like the old flow).
2023-01-11 12:26:09 +01:00
Simon Backx
6a664d11b9 Added 100% test coverage to email renderer and fixed authors bug
refs https://github.com/TryGhost/Team/issues/2339

This fixed a bug in the new email flow that more than 2 authors were displayed as 'undefined & 2 others'.
2023-01-11 12:14:02 +01:00
Simon Backx
6a364e7779 Added 100% test coverage for EmailEventStorage
refs https://github.com/TryGhost/Team/issues/2339
2023-01-10 16:36:41 +01:00
Simon Backx
9ca2e3f183 🐛 Fixed storing email recipient failures
fixes https://github.com/TryGhost/Team/issues/2398

There was an error when fetching the existing email recipient failure. It ended up matching all recipient failures. The result was that only one failure was stored in the database.
2023-01-10 15:41:42 +01:00
Simon Backx
bdf45fad84 Added 100% test coverage to BatchSendingService
refs https://github.com/TryGhost/Team/issues/2339
2023-01-10 14:58:50 +01:00
Elena Baidakova
69d4a96fb8
🎨 Add ability to send test email with chosen newsletter (#15783)
closes TryGhost/Team#1897

-
[Design](https://www.figma.com/file/RpEbPA7H7VHLtXjt3YyB2t/Multiple-Newsletters-Preview?node-id=0%3A1&t=YkDXy063OkCrAI4a-0)
-
[Discussion](https://ghost.slack.com/archives/C019B1K4FAM/p1667924062808939)
2023-01-09 17:48:30 +04:00
Elena Baidakova
5b5f4cdd3f
🐛 Fixed feedback buttons for dark mode (#16091)
refs TryGhost/Team#2396
2023-01-09 16:40:42 +04:00
Simon Backx
c31bc58210 Added 100% test coverage to EmailController
refs https://github.com/TryGhost/Team/issues/2339
2023-01-04 15:22:49 +01:00
Simon Backx
819d0d884c
Improved email verification required checks (#16060)
fixes https://github.com/TryGhost/Team/issues/2366
refs https://ghost.slack.com/archives/C02G9E68C/p1670232405014209

Probem described in issue.

In the old MEGA flow:
- The `email_verification_required` check is now repeated inside the job

In the new email service flow:
- The `email_verification_required` is now checked (didn't happen
before)
- When generating the email batch recipients, we only include members
that were created before the email was created. That way it is
impossible to avoid limit checks by inserting new members between
creating an email and sending an email.
- We don't need to repeat the check inside the job because of the above
changes

Improved handling of large imports:
- When checking `email_verification_required`, we now also check if the
import threshold is reached (a new method is introduced in
vertificationTrigger specifically for this usage). If it is, we start
the verification progress. This is required for long running imports
that only check the verification threshold at the very end.
- This change increases the concurrency of fastq to 3 (refs
https://ghost.slack.com/archives/C02G9E68C/p1670232405014209). So when
running a long import, it is now possible to send emails without having
to wait for the import. Above change makes sure it is not possible to
get around the verification limits.

Refactoring:
- Removed the need to use `updateVerificationTrigger` by making
thresholds getters instead of fixed variables.
- Improved awaiting of members import job in regression test
2023-01-04 11:22:12 +01:00
Rishabh Garg
a09e86da05
Fixed email preview text with new stability flow (#15996)
closes https://github.com/TryGhost/Team/issues/2382

The preview text is getting set to subject line in the new email flow so it repeats multiple times in the inbox(subject+preview+title). This was because the new flow doesn't use the post serialisation that the old system did, causing excerpt to be empty in the email rendering.

Old system was using post serialisation here -
a721e4f2d7/ghost/core/core/server/services/mega/post-email-serializer.js (L136-L139).

This change adds explicit method to calculate the preview text for email in email renderer service using same logic as used in old system.

Co-authored-by: Simon Backx <git@simonbackx.com>
2022-12-14 15:54:26 +05:30
Simon Backx
47cd7a7095
🐛 Handled unknown Mailgun events (#15995)
refs https://ghost.slack.com/archives/C02G9E68C/p1670916538764019

- We receive events that don't have an emailId or providerId.
- We filter those events now and log them as an error
2022-12-14 11:17:45 +01:00
Elena Baidakova
8d2418578d
🐛 Removed horizontal scroll for long author's name (#15985)
closes TryGhost/Team#2272
- Show all name's text due to poor support `text-overflow: ellipsis` in
email clients
2022-12-12 15:36:58 +04:00
Simon Backx
9e6f5e93d8
🐛 Fixed email header images serving original image size (#15950)
fixes https://github.com/TryGhost/Team/issues/2368

- Removed the usage of the `isLocalContentImage` Koenig util for the
email header and feature image url generation.
- While we were trying to set the width to 1200px, we didn't have that
size hardcoded. So that url would redirect back to the original location
instead of serving a smaller image. So I added a new internal size to
the `imageOptimization` config.
- This is fixed in both the new and old email flow and includes some
extra tests for the new flow.
2022-12-09 11:17:22 +01:00
Simon Backx
c47891c3f6
🐛 Fixed setting delivered_at to null after hard bounce (#15942)
refs https://ghost.slack.com/archives/C02G9E68C/p1670075366333929?thread_ts=1669963540.980309&cid=C02G9E68C

When we receive a permanent bounce/failure, we set delivered_at to null.
But we don't want to lose this information.

Instead we should be able to handle recipients that both have failed_at
and delivered_at set.
2022-12-06 10:26:54 +05:30
Simon Backx
2519f4b20d Disabled autoRefresh for batch sending service to increase performance
fixes https://github.com/TryGhost/Team/issues/2363
2022-12-05 12:09:30 +01:00
Simon Backx
ea72934c41
Improved email concurrency sending (#15923)
no issue

Send 10 batches at the same time
2022-12-02 15:30:02 +01:00
Simon Backx
1958d0db63
Removed spaces and quotes from replacement ids (#15921)
fixes https://github.com/TryGhost/Team/issues/2350
2022-12-02 10:49:01 +01:00
Rishabh Garg
f9161876b3
Updated unit test coverage for email service (#15913)
refs https://github.com/TryGhost/Team/issues/2339

- adds unit test for email renderer, segmenter and sending service
2022-12-02 00:33:28 +05:30
Simon Backx
0182965aa3 Updated batch sending to log errors separately
fixes https://github.com/TryGhost/Team/issues/2347

Makes sure we can use the error code
2022-12-01 16:35:54 +01:00
Simon Backx
4c166e11df
Added E2E tests for batch sending (#15910)
refs https://github.com/TryGhost/Team/issues/2339

- Includes a new pattern in the job manager that allows us to properly
await jobs.
- Added new convenience mocking methods to stub settings
- Tests the main flows for bulk sending:
    - Sending in multiple batches
    - Sending to multiple segments
    - Handling a failed batch and retrying that batch
- Fixes bug in batch generation (ordering not working)

In a different PR I'll add more detailed tests.
2022-12-01 13:43:49 +01:00