Commit Graph

71 Commits

Author SHA1 Message Date
Daniel Lockyer
42768937fd Disabled autoRefresh when adding an email recipient failure
- we don't need the updated model once we've saved it in the DB, so we
  can disable the auto refresh in Bookshelf to save a query
2023-02-27 15:45:25 +01:00
Daniel Lockyer
2d1f9fff0c
Updated @tryghost/errors dependency
- there's a weird situation when we have mixed versions of the
  dependency because different libraries try to compare instances
- this brings the usage up to 1.2.21 so we can fix the build for now
2023-02-22 11:32:11 +01:00
Simon Backx
fb210ba6a0 Added extra Mailgun retry
no issue
2023-02-21 16:22:26 +01:00
Simon Backx
7781c2da07 Implemented retrying for sending email via Mailgun API
no issue

Retry sending an email up to 5 times if it failed.
2023-02-21 16:06:23 +01:00
Simon Backx
b665b1a3cc Implemented email analytics retrying (#16273)
fixes https://github.com/TryGhost/Team/issues/2562

New event fetching loops:
- Reworked the analytics fetching algorithm. Instead of starting again
where we stopped during the last fetching minus 30 minutes, we now just
continue where we stopped. But with ms precision (because no longer
database dependent after first fetch), and we stop at NOW - 1 minute to
reduce chance of missing events.
- Apart from that, a missing fetching loop is introduced. This fetches
events that are older than 30 minutes, and just processes all events a
second time to make sure we didn't skip any because of storage delays in
the Mailgun API.
- A new scheduled fetching loop, that allows us to schedule between a
given start/end date (currently only persisted in memory, so stops after
a reboot)

UI and endpoint changes:
- New UI to show the state of the analytics 'loops'
- New endpoint to request the analytics loop status
- New endpoint to schedule analytics
- New endpoint to cancel scheduled analytics
- Some number formatting improvements, and introduction of 'opened'
count in debug screen
- Live reload of data in the debug screen

Other changes:
- This also improves the support for maxEvents. We can now stop a
fetching loop after x events without worrying about lost events. This is
used to reduce the fetched events in the missing and scheduled event
loop (e.g. when the main one is fetching lots of events, we skip the
other loops).
- Prevents fetching the same events over and over again if no new events
come in (because we always started at the same begin timestamp). The
code increases the begin timestamp with 1 second if it is safe to do so,
to prevent the API from returning the same events over and over again.
- Some optimisations in handing the processing results (less merges to
reduce CPU usage in cases we have lots of events).

Testing:
- You can test with lots of events using the new mailgun mocking server
(Toolbox repo `scripts/mailgun-mock-server`). This can also simulate
events that are only returned after x minutes because of storage delays.
2023-02-21 16:00:32 +01:00
renovate[bot]
cf7ecb492c
Update dependency c8 to v7.13.0 2023-02-16 22:15:50 +00:00
Simon Backx
8dfa490638
Added outbound link tagging for web posts (#16201)
fixes https://github.com/TryGhost/Team/issues/2433

- Moved all outbound link tagging code to separate OutboundLinkTagger
- Because a site can easily enable/disable this feature, we don't store
the ?refs in the HTML but add them on the fly for now in the Content
API.
2023-02-16 11:26:35 +01:00
Peter Zimon
5bb5977a47 Fixed issue with bookmark cards width on Android
no refs.

- On Android Gmail client the bookmark cards were not taking the full width of the screen
2023-02-14 14:31:16 +01:00
Simon Backx
0e8d13bdde 🐛 Fixed email replacements without fallback
fixes https://github.com/TryGhost/Team/issues/2557

When a member doen't have a name, and the first_name replacement doesn't have a fallback, we did show %recipient.first_name% instead of an empty string.
2023-02-13 15:58:40 +01:00
Simon Backx
48f9485f46
🐛 Fixed storing email failures with an empty message (#16260)
no issue

- When we receive an email failure with an empty message, the saving of
the model would fail because of schema validation that requires strings
to be non-empty.
- This adds more logging to the email analytics service to help debug
future issues
- Performance improvement to storing delivered, opened and failed emails
by replacing COALESCE with WHERE X IS NULL (tested and should give a
decent performance boost locally).
2023-02-13 15:25:36 +01:00
Simon Backx
ea2c69565f
Moved email event fetching to main thread (#16231) 2023-02-09 09:36:39 +01:00
Simon Backx
fb3cbe5fc8
Added short lived caching to email batch body (#16233)
fixes https://github.com/TryGhost/Team/issues/2522

When sending an email for multiple batches at the same time, we now
reuse the same email body for each batch in the same segment. This
reduces the amount of database queries and makes the sending more
reliable in case of database failures.

The cache is short lived. After sending the email it is automatically
garbage collected.
2023-02-07 11:01:49 +01:00
Simon Backx
30b9d02f70 Increased wait time between Mailgun events to 200ms
no issue

Reduce concurrency if the database is going a bit slower, until we have more permanent fix for this.
2023-02-03 14:38:44 +01:00
Simon Backx
29e1e93ca1 Increased wait time between Mailgun events to 150ms
no issue

Reduce concurrency if the database is going a bit slower, until we have more permanent fix for this.
2023-02-03 10:39:55 +01:00
Simon Backx
4fad1986a0 🐛 Fixed link to posts on audio cards in emails
refs https://ghost.slack.com/archives/CTH5NDJMS/p1675356984043299

The url property doesn't exists on post, this was copied from the old MEGA flow that serialized the model first.
2023-02-03 10:37:02 +01:00
Simon Backx
06de10786f
Added automatic database query retrying to the email service (#16218)
fixes https://github.com/TryGhost/Team/issues/2512

The email sending is a crucial flow that should not be interrupted. If
there are database connection issues, we should try to recover from them
automatically by retrying individual database queries and transactions
for a limited amount of time.

This adds a helper method to retry database operations. It limits all
database queries before sending the actual email to maximum 10 minutes.
Database operations that happen after sending have a higher retry time
because they are more crucial to prevent data loss (e.g. saving that an
email was sent).
2023-02-02 14:12:54 +01:00
Simon Backx
b50e3915ea 🐛 Fixed HTML escaping of feature_image_caption in newsletters
no issue

feature_image_caption was escaped in the new email stability flow, while that should not happen (bold/underline/...).
2023-01-30 16:58:59 +01:00
Simon Backx
c798f383f9 🐛 Fixed not reactivating email analytics jobs
no issue

When a site doesn't have any emails on boot, it doesn't schedule the email analytics job. With this change, the new email flow will also restart that job after an email has been created.
2023-01-26 17:35:45 +01:00
Simon Backx
5e1e6c9863
Improved email error logging (#16184)
no issue

Logs errors to Sentry and adds error codes.
2023-01-25 14:57:10 +01:00
Simon Backx
f4c55d123c
🐛 Fixed email segment generation (#16182)
fixes https://github.com/TryGhost/Team/issues/2484

The flow only send the email to segments that were targeted in the email
content. But if a part of the email is only visible for `status:free`,
that doesn't mean we don't want to send the email to `status:-free`.
This has been corrected in the new email flow.
2023-01-25 14:56:37 +01:00
Simon Backx
4b0ca9399d 🐛 Reduced concurrency when fetching Mailgun events (#16176)
refs https://github.com/TryGhost/Team/issues/2482

This change adds a small sleep in between dispatching events in the
worker thread that reads the events from Mailgun. That should reduce the
amount of queries we fire parallel to each other and could cause the
connection pool to run out of connections.

It also reduces the amount of concurrent sending to 2 from 10. Also to
make sure the connection pool doesn't run out of connections while
sending emails, and to reduce the chance of new connections falling back
on a (delayed) replicated database.
2023-01-25 13:07:01 +01:00
Rishabh Garg
88979c852b
Updated email sending to remove invalid recipient emails (#16171)
closes https://github.com/TryGhost/Team/issues/2388

We have seen examples of sites with member emails that have invalid characters that can cause an entire email send to fail, or just cause a failure to those addresses. The issue that allowed members with invalid email address to be saved was patched earlier, but its possible there are still sites that contain some of those invalid email addresses.

This change updates new sending service to filter out the recipients with invalid email address before passing them to mail provider, so these rogue addresses don't affect the whole batch in anyway. We also trim the recipient emails to clear out any spaces first, which is the most likely culprit.

- uses new email validator that detects invalid email addresses with special chars
2023-01-24 16:13:10 +05:30
Sanne de Vries
4b61f23bd0 Fixed minor email preview visual bugs
No ref
2023-01-23 10:46:30 +01:00
Simon Backx
3ac282598c Fixed Playwright tests for new email flow
no issue

Tests stopped working because the Mailgun mocker stopped working since we moved to the new email flow.

This also fixes a unit test that needed to get updated.
2023-01-20 19:00:13 +01:00
Simon Backx
693216c29e Improved failed email error messages
refs https://ghost.slack.com/archives/C02G9E68C/p1673570945616369?thread_ts=1672700158.659209&cid=C02G9E68C

Fixed a typo in the error messages and made the errors more clear. Also hide the email configuration part when there is no email configuration.
2023-01-20 18:37:43 +01:00
Simon Backx
e879406659
Added outbound link tagging setting (#16146)
fixes https://github.com/TryGhost/Team/issues/2432
    
Adds outbound_link_tagging setting (enabled by default and behind
feature flag). If the feature flag is enabled, and the setting is
disabled, we won't add ?ref to links in emails.
    
This includes new E2E tests for email click tracking, which were also
extended to check outbound link tagging (for both MEGA and the new email
stability flow).

Also fixes a test fixture for the comments_enabled setting.
2023-01-20 13:41:36 +01:00
Daniel Lockyer
34fe49b110
Added version information to log lines
refs https://github.com/TryGhost/Toolbox/issues/501

- this reverts commit 48dda23554
- also includes a resolution for `@elastic/elasticsearch` so we don't
  run a version that is potentially problematic - see referenced issue
  for context
2023-01-20 13:18:44 +01:00
Daniel Lockyer
48dda23554
Reverted "Added version information to log lines"
refs https://github.com/TryGhost/Toolbox/issues/501

- this reverts commit f2116357b7
- something with Elasticsearch is causing high CPU usage, so this commit
  reverts that for now
2023-01-17 13:20:47 +01:00
Sam Lord
f2116357b7 Added version information to log lines
refs: https://github.com/TryGhost/Toolbox/issues/502

Updated to @tryghost/logging@2.4.0 to allow metadata to be logged
2023-01-12 10:34:23 +00:00
Simon Backx
d5a359f2db Removed possibility to use {uuid} replacements in email
fixes https://github.com/TryGhost/Team/issues/2383

A user could use `{uuid}` inside an email only content and it would work. This currently isn't supposed to be used outside internal features (link click tracking, feedback buttons). For now this is only fixed in the new email flow under the email stability flag.
2023-01-11 15:56:15 +01:00
Simon Backx
8a2303413d Restricted email-service package to 100% test coverage
fixes https://github.com/TryGhost/Team/issues/2339

The email service is now fully covered by tests, and this commit also forces the test coverage to remain 100% after future changes.
2023-01-11 13:54:26 +01:00
Simon Backx
ee36a0fce9 Added 100% test coverage to SendingService
refs https://github.com/TryGhost/Team/issues/2339
2023-01-11 13:39:25 +01:00
Simon Backx
9650521376 Fixed variable date formatting in new email flow
refs https://github.com/TryGhost/Ghost/actions/runs/3892250013/jobs/6643448457

Depending on the environment the date formatting in a newsletter could change. Now it is always set to the DD MMM YYYY format (like the old flow).
2023-01-11 12:26:09 +01:00
Simon Backx
6a664d11b9 Added 100% test coverage to email renderer and fixed authors bug
refs https://github.com/TryGhost/Team/issues/2339

This fixed a bug in the new email flow that more than 2 authors were displayed as 'undefined & 2 others'.
2023-01-11 12:14:02 +01:00
Simon Backx
6a364e7779 Added 100% test coverage for EmailEventStorage
refs https://github.com/TryGhost/Team/issues/2339
2023-01-10 16:36:41 +01:00
Simon Backx
9ca2e3f183 🐛 Fixed storing email recipient failures
fixes https://github.com/TryGhost/Team/issues/2398

There was an error when fetching the existing email recipient failure. It ended up matching all recipient failures. The result was that only one failure was stored in the database.
2023-01-10 15:41:42 +01:00
Simon Backx
bdf45fad84 Added 100% test coverage to BatchSendingService
refs https://github.com/TryGhost/Team/issues/2339
2023-01-10 14:58:50 +01:00
Elena Baidakova
69d4a96fb8
🎨 Add ability to send test email with chosen newsletter (#15783)
closes TryGhost/Team#1897

-
[Design](https://www.figma.com/file/RpEbPA7H7VHLtXjt3YyB2t/Multiple-Newsletters-Preview?node-id=0%3A1&t=YkDXy063OkCrAI4a-0)
-
[Discussion](https://ghost.slack.com/archives/C019B1K4FAM/p1667924062808939)
2023-01-09 17:48:30 +04:00
Elena Baidakova
5b5f4cdd3f
🐛 Fixed feedback buttons for dark mode (#16091)
refs TryGhost/Team#2396
2023-01-09 16:40:42 +04:00
Simon Backx
30853bc213 Added 100% test coverage to EmailEventProcessor
refs https://github.com/TryGhost/Team/issues/2339
2023-01-04 16:07:19 +01:00
Simon Backx
c31bc58210 Added 100% test coverage to EmailController
refs https://github.com/TryGhost/Team/issues/2339
2023-01-04 15:22:49 +01:00
Simon Backx
e2e9a56583 Added 100% test coverage to EmailService class
refs https://github.com/TryGhost/Team/issues/2339

Tests the EmailService class of the email-service package.
2023-01-04 14:25:29 +01:00
Daniel Lockyer
aa08fc72e7 Updated Koenig packages
refs https://github.com/TryGhost/Toolbox/issues/488

- these packages add support for Node 18 and drop support for Node 12
2023-01-04 11:27:39 +01:00
Simon Backx
819d0d884c
Improved email verification required checks (#16060)
fixes https://github.com/TryGhost/Team/issues/2366
refs https://ghost.slack.com/archives/C02G9E68C/p1670232405014209

Probem described in issue.

In the old MEGA flow:
- The `email_verification_required` check is now repeated inside the job

In the new email service flow:
- The `email_verification_required` is now checked (didn't happen
before)
- When generating the email batch recipients, we only include members
that were created before the email was created. That way it is
impossible to avoid limit checks by inserting new members between
creating an email and sending an email.
- We don't need to repeat the check inside the job because of the above
changes

Improved handling of large imports:
- When checking `email_verification_required`, we now also check if the
import threshold is reached (a new method is introduced in
vertificationTrigger specifically for this usage). If it is, we start
the verification progress. This is required for long running imports
that only check the verification threshold at the very end.
- This change increases the concurrency of fastq to 3 (refs
https://ghost.slack.com/archives/C02G9E68C/p1670232405014209). So when
running a long import, it is now possible to send emails without having
to wait for the import. Above change makes sure it is not possible to
get around the verification limits.

Refactoring:
- Removed the need to use `updateVerificationTrigger` by making
thresholds getters instead of fixed variables.
- Improved awaiting of members import job in regression test
2023-01-04 11:22:12 +01:00
Daniel Lockyer
08b786af3b Bumped TryGhost-owned dependencies and lockfile
- this was all getting terribly behind so I've done several things:
  - majority of `@tryghost/*` except Lexical packages
  - gscan + knex-migrator to remove old `@tryghost/errors` usage
  - bumped lockfile
2023-01-02 20:55:22 +01:00
Rishabh Garg
a09e86da05
Fixed email preview text with new stability flow (#15996)
closes https://github.com/TryGhost/Team/issues/2382

The preview text is getting set to subject line in the new email flow so it repeats multiple times in the inbox(subject+preview+title). This was because the new flow doesn't use the post serialisation that the old system did, causing excerpt to be empty in the email rendering.

Old system was using post serialisation here -
a721e4f2d7/ghost/core/core/server/services/mega/post-email-serializer.js (L136-L139).

This change adds explicit method to calculate the preview text for email in email renderer service using same logic as used in old system.

Co-authored-by: Simon Backx <git@simonbackx.com>
2022-12-14 15:54:26 +05:30
Simon Backx
47cd7a7095
🐛 Handled unknown Mailgun events (#15995)
refs https://ghost.slack.com/archives/C02G9E68C/p1670916538764019

- We receive events that don't have an emailId or providerId.
- We filter those events now and log them as an error
2022-12-14 11:17:45 +01:00
Daniel Lockyer
6f4e663d74
Updated @tryghost dependencies (#16005)
- also includes `knex-migrator` with a simple `sqlite3` bump
2022-12-14 11:18:55 +07:00
renovate[bot]
13abcf6c9d
Update dependency mocha to v10.2.0 2022-12-12 13:20:22 +00:00
Elena Baidakova
8d2418578d
🐛 Removed horizontal scroll for long author's name (#15985)
closes TryGhost/Team#2272
- Show all name's text due to poor support `text-overflow: ellipsis` in
email clients
2022-12-12 15:36:58 +04:00