Commit Graph

192 Commits

Author SHA1 Message Date
Kevin Ansfield
a87410ef28 🐛 Fixed logs showing an error when sending an email with no feature image
no issue

- we were attempting to read an image file to determine it's dimensions when no feature image was set. This wasn't a fatal error as it was handled gracefully and had no ill consequences but it was adding confusing errors to the logs
2021-06-01 17:07:33 +01:00
Kevin Ansfield
84724537be 🐛 Fixed feature images in emails appearing very wide in Outlook
closes https://github.com/TryGhost/Team/issues/675

Outlook will display images at their native resolution if no `width` attribute is supplied. Content images were fixed a while ago but feature images would still render very wide and cause horizontal scroll and text size/alignment issues.

- modify `post.feature_image` and add a `post.feature_image_width` property before passing it through to the email template
  - for Unsplash images we assume all images are larger than 600px so we change the URL to reference a 1200px image and set the image width to 600 (to keep images on retina displays crisp)
  - for other images we probe the image to fetch the original dimensions and give set an image width of 600 if needed, if it's a locally-hosted image we update the URL to point at a max 1200px version
- updated email template to output a `width` attribute on the feature image `<img>` tag if it's set
2021-05-14 11:57:37 +01:00
Naz
fc458efa90 Added periodic "emails" limit check when (re)sending a newsletter
refs https://github.com/TryGhost/Team/issues/588
refs d72ba77aba

- When limit is in place we don't want to allow sending out a new batch of emails if it would go over limit
- See referenced commit for example configuration
2021-05-07 18:45:14 +04:00
Kevin Ansfield
322664a145
Added ability to send a newsletter to members with a certain label or product (#12932)
refs https://github.com/TryGhost/Team/issues/581
refs https://github.com/TryGhost/Team/issues/582

When publishing a post via the API it was possible to send it using `?email_recipient_filter=all/free/paid` which allowed you to send to members only based on their payment status which is quite limiting for some sites.

This PR updates the `?email_recipient_filter` query param to support Ghost's `?filter` param syntax which enables more specific recipient lists, eg:

`?email_recipient_filter=status:free` = free members only
`?email_recipient_filter=status:paid` = paid members only
`?email_recipient_filter=label:vip` = members that have the `vip` label attached
`?email_recipient_filter=status:paid,label:vip` = paid members and members that have the `vip` label attached

The older `free/paid` values are still supported by the API for backwards compatibility.

- updates `Post` and `Email` models to transform legacy `free` and `paid` values to their NQL equivalents on read/write
  - lets us not worry about supporting legacy values elsewhere in the code
  - cleanup migration to transform all rows slated for 5.0
- removes schema and API `isIn` validations for recipient filters so allow free-form filters
- updates posts API input serializers to transform `free` and `paid` values in the `?email_recipient_filter` param to their NQL equivalents for backwards compatibility
- updates Post API controllers `edit` methods to run a query using the supplied filter to verify that it's valid
- updates `mega` service to use the filter directly when selecting recipients
2021-05-07 11:56:41 +01:00
Hannah Wolfe
273e220327 Moved i18n to shared
refs 829e8ed010

- i18n is used everywhere but only requires shared or external packages, therefore it's a good candidate for living in shared
- this reduces invalid requires across frontend and server, and lets us use it everywhere until we come up with a better option
2021-05-04 13:03:38 +01:00
Hannah Wolfe
829e8ed010 Expanded requires of lib/common i18n and events
- Having these as destructured from the same package is hindering refactoring now
- Events should really only ever be used server-side
- i18n should be a shared module for now so it can be used everywhere until we figure out something better
- Having them seperate also allows us to lint them properly
2021-05-03 17:14:52 +01:00
Naz
bc15f8c1bb Added "emails" limit check when (re)sending a newsletter
refs https://github.com/TryGhost/Team/issues/588

- This check allows for a on/off switch to be set up on the instance and control limits around sending emails
- An example configuration for such check would look like following in config's hostSettings section, e.g.:
```
"emails": {
    "disabled": true,
    "error": "Email sending has been temporarily disabled whilst your account is under review."
}
```
2021-05-03 17:51:33 +04:00
Daniel Lockyer
e9b21fdbd1 Updated bson-objectid calls to match API change
refs c873899e49

- as of `bson-objectid` v2.0.0, this library exports the function
  to generate an ObjectID directly, and then you need to use `.toHexString()`
  to get the 24 character hex string - 6696f27d82
- this commit removes all uses of `.generate()` and replaces with this
  change
2021-04-21 16:23:52 +01:00
Peter Zimon
30bc751237 Updated 'Powered by' badge ref to 4.0 repo 2021-03-10 16:02:00 +01:00
Hannah Wolfe
d51fcd072e Added new member limit & removed old
refs: https://github.com/TryGhost/Team/issues/510

- The current member limit was implemented as a member-specific concept
- The new limit service is much more generic, here we are swapping old for new
- The updated concept here is blocking all publishing, not just email sending, when a site is over its member limit
- To determine that we are publishing a post, we must be in the model layer. The code has been moved to the permissible function which makes sense as this is a permissions error that we are throwing
- I've left the extra check for email retries in, in case there is some loophole here (but we may wish to change it)
2021-03-04 18:16:34 +00:00
Naz
b3542e9176 Added JSDocs to MEGA service
refs d5cf0fc03e

- Makes sure required `apiVerson` parameter in MEGA is visible and easy to discover
2021-03-02 14:32:43 +13:00
Naz
d5cf0fc03e Bumped MEGA related serialization code to use v4 API
refs https://github.com/TryGhost/Ghost/issues/12716

- The code in serializePostModel was broken and always defaulted to 'v3'!  It refered to non-existent `model.get('api_version')` there's no such field in posts model! Changed the implementation so that the API version is passed in as a parameter to the method instead
- The style of providing "defaults" everywhere creates a need for future maintenance when we bump the version e.g in Ghost v5. Maybe reworking these methods to require a passed version and throwing an error instead would be more maintainable long-term?
2021-03-02 14:31:01 +13:00
Fabien 'egg' O'Carroll
f4cb5c57c6
Updated members_status_events table (#12647)
refs https://github.com/TryGhost/Ghost/issues/12602

* Updated members_status_events table

By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range

* Updated MEGA to handle addition of 'comped' status

With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"

* Updated customQuery for MemberStatusEvent

With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.

* Populated members_status_events table

As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.

The assumptions used in backfilling is that a Member's current status,
is their only status.
2021-02-16 10:38:36 +00:00
Sanne de Vries
67b1745bcb Fixed bug for newsletter bookmarks without author displaying redundant separator 2021-02-09 09:35:45 +01:00
Sanne de Vries
118ceae29c Fixed gmail bug for newsletter bookmark cards not displaying correctly 2021-02-08 16:49:13 +01:00
Fabien 'egg' O'Carroll
6af2706f10
Updated Admin API and Mega to use status flag (#12579)
no-issue

* Removed support for paid param from v3 & canary API
* Updated active subscription checks to use status flag
* Updated MEGA to use status filter over paid flag
* Removed support for paid option at model level
* Installed @tryghost/members-api@1.0.0-rc.0
* Updated members fixtures
2021-02-02 16:12:51 +00:00
Sanne de Vries
c8931a47b5 Updated newsletter blockquote font styling 2021-01-15 15:42:01 +01:00
Rish
a1663bea91 💄Updated newsletter design to use accent color
no issue

- Updates newsletter design to use accent color for blockquote border if present
2021-01-15 19:47:32 +05:30
Naz
b2e7d2bf06 Bumped job-manager version to 0.7.0
closes https://github.com/TryGhost/Ghost-Utils/issues/122
2021-01-06 17:48:05 +13:00
Daniel Lockyer
947603d0e3
Added original error to thrown error in mega code
- the original error is not propagated up with the error we throw, so it
  can sometimes be difficult to see what's going on down the line
2020-12-02 09:04:20 +00:00
Kevin Ansfield
d675278b0b
Prevented scheduling of recurring analytics jobs when not using emails (#12441)
no issue

- recurring jobs spin up worker threads which can be quite CPU intensive even when not performing much processing, this can be problematic in environments where there are many Ghost instances running
- updated the email job scheduling to be skipped on bootup when there are no emails in the database and to be started when the first email is created as long as we're not in testing env
- increase analytics job schedule from every 2 minutes to every 5 minutes to help spread the load further across instances
2020-12-02 08:17:44 +00:00
Kevin Ansfield
47e81e3ca1 Fixed error when creating emails when track opens is disabled
no issue

- ensure `email.track_opens` is a boolean rather than `null`
2020-11-25 08:08:15 +00:00
Kevin Ansfield
7bbbeeb5b0 Fixed linting 2020-11-23 19:58:34 +00:00
Kevin Ansfield
a8d8598003 Updated mega service to use open tracking from settings
refs https://github.com/TryGhost/Ghost/issues/12390

- switched from enabling open tracking based on `enableDeveloperExperiments` to using the track-opens setting
2020-11-23 18:53:55 +00:00
Peter Zimon
bf9820f97d Fixed sans-serif link style
no refs.
- fixed regression that links in emails with sans-serif setting wasn't underlined
2020-11-23 15:50:02 +01:00
Peter Zimon
02d78a89aa Udpated email template footer spacings
no refs.
- updated padding between Ghost badge and rest of the footer
2020-11-16 12:31:37 +01:00
Peter Zimon
cfac18cb32 Newsletter email template refinements
no refs.
- updated color styles, links for footer text
- refined spacings for email footer paragraphs
2020-11-13 16:58:56 +01:00
Fabien 'egg' O'Carroll
3f9053ae94
Added setting for newsletter footer content (#12359)
no-issue

We want to give users to ability to customise the content of their newsletter, and the first step
toward that is a setting in which we can store text or html to embed in the template
2020-11-13 13:18:47 +00:00
Peter Zimon
215bfd0a7a
Added newsletter design settings (#12352)
refs https://github.com/TryGhost/Ghost/issues/12355

- Adds new default settings for newsletter customisations - `newsletter_show_badge`, `newsletter_show_header` and `newsletter_body_font_category`
- Adds migrations to update group for new settings
- Add migration to update settings based on existing config value for newsletter settings
- Passes new newsletter settings to newsletter template and updates design based on them
- Fix tests
2020-11-11 18:26:11 +05:30
Fabien 'egg' O'Carroll
6140a98351
Updated newsletter functionality to use email_recipient_filter (#12343)
no-issue

* Used email_recipient_filter in MEGA

This officially decouples the newsletter recipients from the post
visibility allowing us to send emails to free members only

* Supported enum for send_email_when_published in model

This allows us to migrate from the previously used boolean to an enum
when we eventually rename the email_recipient_filter column to
send_email_when_published

* Updated the posts API to handle email_recipient_filter

We now no longer rely on the send_email_when_published property to send
newsletters, meaning we can remove the column and start cleaning up the
new columns name

* Handled draft status changes when emails not sent

We want to reset any concept of sending an email when a post is
transition to the draft status, if and only if, and email has not
already been sent. If an email has been sent, we should leave the email
related fields as they were.

* Removed send_email_when_published from add method

This is not supported at the model layer

* Removed email_recipient_filter from v2&Content API

This should not be exposed on previous api versions, or publicly

* Removed reference to send_email_when_published

This allows us to move completely to the email_recipient_filter
property, keeping the code clean and allowing us to delete the
send_email_when_published column in the database. We plan to then
migrate _back_ to the send_email_when_published name at both the
database and api level.
2020-11-06 17:32:23 +00:00
Fabien O'Carroll
44b43a4755 Fixed error with removal of global.Promise override
refs #12182

These were missed in the cleanup and were causing errors.
2020-11-06 16:49:10 +00:00
Kevin Ansfield
aface9ed4c Enabled Mailgun open tracking when dev experiments is enabled
no issue

- set `emails.track_opens` to `true` when the `enableDeveloperExperiments` flag is set
- update mailgun bulk-email provider to pass the open-tracking header to Mailgun when the email's `track_opens` flag is set
2020-11-05 12:55:08 +00:00
Kevin Ansfield
4f211d025d Fixed members with multiple subscriptions receiving multiple newsletters
closes https://github.com/TryGhost/Ghost/issues/12259

- adds a `DISTINCT` to the query used to fetch member rows when generating an email recipient list
- this increases query time 2.7s vs 1.6s locally with ~94k paid members but once the `members.paid` column is implemented this slow query can be removed
2020-10-05 16:53:35 +01:00
Kevin Ansfield
b5ffb38726 Fixed partial email batch/recipient records when email creation fails
no issue

- wrap email batch/recipient record creation in a transaction so if an error occurs during creation we're not left with a partially created batch/recipient set in the database
2020-10-02 13:47:14 +01:00
Kevin Ansfield
0f30b9f6a0 🐛 Fixed email not showing as failed if error occurs when preparing email
no issue

- if an error occurred whilst creating email batch/recipient records the email status was never updated and was left in the 'pending' status
- adjusted the error handling to update the email status and record the error message if such a scenario occurs
2020-10-02 12:40:49 +01:00
Kevin Ansfield
7b789e1cd5 🐛 Fixed newsletters being sent to Stripe customer emails in place of member emails
no issue

- the paid-member SQL query that is obtained using `models.Member.getFilteredCollectionQuery({paid: true})` can return multiple columns with the same name (eg, `email`, `name`), when that happens the last column with duplicate names "wins" and it's value is used in the resulting knex row instance
- in the `mega` service when fetching email recipient rows we ran into this problem, to avoid it we adjust the query to explicitly select only the data from the `members` table
2020-10-02 12:17:17 +01:00
Kevin Ansfield
82126f29e6 Added guard for member rows having missing data when creating email recipients
no issue

- we've had an issue with emails failing due to unexpectedly missing data when inserting email recipient rows
- added a validation check before adding recipient details along with a log so that invalid data can be investigated
2020-10-02 11:29:10 +01:00
Kevin Ansfield
474e6c4c45 Refactor mega service to use stored email content and batch/recipient records
no issue

- store raw content in email record
  - keep any replacement strings in the html/plaintext content so that it can be used when sending email rather than needing to re-serialize the post content which may have changed
- split post email serializer into separate serialization and replacement parsing functions
  - serialization now returns any email content that is derived from the post content (subject/html/plaintext) rather than post content plus replacements
  - `parseReplacements` has been split out so that it can be run against email content rather than a post, this allows mega and the email preview service to work with the stored email content
- move mailgun-specific functionality into the mailgun provider
  - previously mailgun-specific behaviour was spread across the post email serializer, mega, and bulk-email service
  - the per-batch `send` functionality was moved from the `bulk-email` service to the mailgun provider and updated to take email content, recipient info, and replacement info so that all mailgun-specific headers and replacement formatting can be handled in one place
  - exposes the `BATCH_SIZE` constant because batch sizes are limited to what the provider allows
- `bulk-email` service split into three methods
  - `send` responsible for taking email content and recipients, parsing replacement info from the email content and using that to collate a recipient data object, and finally coordinating the send via the mailgun provider. Usable directly for use-cases such as test emails
  - `processEmail` takes an email ID, loads it and coordinates sending related batches concurrently
  - `processEmailBatch` takes an email_batch ID, loads it along with associated email_recipient records and passes the data through to the `send` function, updating the batch status as it's processed
  - `processEmail` and `processEmailBatch` take IDs rather than objects ready for future use by job-queues, it's best to keep job parameters as minimal as possible
- refactored `mega` service
  - modified `getEmailData` to collate email content (from/reply-to/subject/html/plaintext) rather than being responsible for dealing with replacements and mailgun-specific replacement formats
    - used for generating email content before storing in the email table, and when sending test emails
    - from/reply-to calculation moved out of the post-email-serializer into mega and extracted into separate functions used by `getEmailData`
  - `sendTestEmail` updated to generate `EmailRecipient`-like objects for each email address so that appropriate data can be supplied to the updated `bulk-email.send` method
  - `sendEmailJob` updated to create `email_batches` and associated `email_recipients` records then hand over processing to the `bulk-email` service
  - member row fetching extracted into a separate function and used by `createEmailBatches`
  - moved updating of email status from `mega` to the `bulk-email` service, keeps concept of Successful/FailedBatch internal to the `bulk-email` service
2020-09-29 17:17:54 +01:00
Kevin Ansfield
d34a3263e8 Store email batch and recipient records when sending newsletters (#12195)
requires https://github.com/TryGhost/Ghost/pull/12192

- added initial `EmailBatch` and `EmailRecipient` model definitions with defaults and relationships
- added missing `post` relationship function to email model
- fetch member list without bookshelf
    - bookshelf can add around 3x overhead when fetching the members list for an email
    - we don't need full members at this point, only having the data is fine
    - if we need full models later on we can push the model hydration into background jobs where recipient batches are fetched ready for an email to be sent
    - bookshelf model instantiation of many models blocks the event loop, using knex directly keeps concurrent requests fast
    - adds `getFilteredCollectionQuery` method to base model to facilitate getting a knex query based on our normal model filters along with transaction/forUpdate applied
- store recipient list before sending email
    - chunk already-fetched members list into batches and insert records into the `email_recipients` table via knex
    - chunked into batches of 1000 to match the number of emails that Mailgun accepts in a single API request but this may not be the absolute fastest batch size for recipient insertion:
        | Batch size | Batch time | Total time |
        | ---------- | ---------- | ---------- |
        |        500 |       20ms |     4142ms |
        |       1000 |       50ms |     4651ms |
        |       5000 |      170ms |     3540ms |
        |      10000 |      370ms |     3684ms |
    - create an email_batch record before inserting recipient rows so we can effeciently fetch recipients by batch and store the overall batch status
2020-09-29 17:17:54 +01:00
Kevin Ansfield
6dc8d91ace Revert "Store email batch and recipient records when sending newsletters (#12195)"
This reverts commit 80af56b530.

- reverting temporarily so that all associated functionality can be merged in a single release
- creating email batch/recipient records without using them would cause inconsistent data
2020-09-21 17:02:59 +01:00
Kevin Ansfield
80af56b530
Store email batch and recipient records when sending newsletters (#12195)
requires https://github.com/TryGhost/Ghost/pull/12192

- added initial `EmailBatch` and `EmailRecipient` model definitions with defaults and relationships
- added missing `post` relationship function to email model
- fetch member list without bookshelf
    - bookshelf can add around 3x overhead when fetching the members list for an email
    - we don't need full members at this point, only having the data is fine
    - if we need full models later on we can push the model hydration into background jobs where recipient batches are fetched ready for an email to be sent
    - bookshelf model instantiation of many models blocks the event loop, using knex directly keeps concurrent requests fast
- store recipient list before sending email
    - chunk already-fetched members list into batches and insert records into the `email_recipients` table via knex
    - chunked into batches of 1000 to match the number of emails that Mailgun accepts in a single API request but this may not be the absolute fastest batch size for recipient insertion:
        | Batch size | Batch time | Total time |
        | ---------- | ---------- | ---------- |
        |        500 |       20ms |     4142ms |
        |       1000 |       50ms |     4651ms |
        |       5000 |      170ms |     3540ms |
        |      10000 |      370ms |     3684ms |
    - create an email_batch record before inserting recipient rows so we can effeciently fetch recipients by batch and store the overall batch status
2020-09-14 15:40:00 +01:00
Rish
dd6ac57aca Fixed missing domain for default support address
no issue

- By default for new sites, support address is set same as from address to `noreply` , with full email address using the domain for `@`
- For newsletter emails, the support address was missing the default site domain to be added to address if its `noreply`
- Fix updates the support address to use the same format as from address and add relevant domain for default case
2020-09-03 16:34:47 +05:30
Rish
696e60dd51 🐛 Fixed missing member email on unsubscribe page
no refs

- The `update` method in members-api package was edited to return Model object instead of JSON directly - TryGhost/Members@a28bcc5
- This unsubscribe handler was returning the raw member object returned from `update` method, which is now a model object and not able to access `member.email`
- Fix updates the unsubscribe request handler to return the member JSON again
2020-09-02 12:03:10 +05:30
Kevin Ansfield
e07e0a17c6 🐛 Fixed bookmark card display in member emails when using Gmail+Chrome
no issue

- updates `@tryghost/kg-default-cards` which contains two fixes
  - removes email-specific output being added to post html (had no visual impact due to use of conditional comments but keeps rendered html smaller+cleaner)
  - adds a background-url style to the thumbnail container to give two options for styling
- updates member email template styling to hide the `<img>` element in bookmark cards and use a background image instead to get consistent rendering across email clients
2020-08-24 18:21:09 +01:00
Peter Zimon
75f5ab637d Added minor member newsletter customisations via config
no issue
- adds a `members:emailTemplate` config object
  - `showSiteHeader` - defaults to `true`, shows the site title and icon in member emails
  - `showPoweredBy` - defaults to `false`, adds a "Publish with Ghost" button to member email footer
- updates member newsletter email template with hideable site header and "powered by" badge
2020-08-24 10:44:22 +01:00
Nazar Gargol
91d1417df2 Fixed email newsletter error when post has no content
closes #12158
2020-08-24 19:46:46 +12:00
Kevin Ansfield
d8e319af88 🐛 Fixed card spacing and caption styling for member emails in Outlook
closes https://github.com/TryGhost/Ghost/issues/12139

- once the email content has been rendered in the post serializer, perform some whole-content transformation of `figure` and `figcaption` to `div` using cheerio
- juiced will have already inlined the elements styles so there's no need to adjust the template's stylesheet
2020-08-19 12:34:14 +01:00
Jeremy Davidson
681c96b5e9
🐛 Fixed bookmark card author/publisher spacing in member emails (#12134)
closes #12078

- Root cause was that pseudo class .kg-bookmark-author:after was not getting inlined to email newsletter or its preview.
  - That is where the margin and bullet-point content are added between author and publisher.
- Dependency juice supports an option inlinePseudoElements which is false by default.
- Fix was to set inlinePseudoElements to true when serializing post email.

Co-authored-by: Jeremy Davidson <jeremy@crossingcontour.com>
2020-08-18 16:18:44 +01:00
Kevin Ansfield
cefcdad090 🐛 Fixed emails appearing very wide in Outlook and improved email image sizes
closes https://github.com/TryGhost/Ghost/issues/11536

- bumps `@tryghost/kg-default-cards`
  - image and gallery cards now output `width/height` attributes on `img` elements with a max width of 600px
  - uses resized images where possible to keep email weight down
- adds `height: auto` style to image card images so that the `height` attribute does not cause distortion at smaller screen widths
2020-08-18 15:33:54 +01:00
Kevin Ansfield
c434666ba2 🐛 Removed [http://url/] output in member email preview text
no issue

- we output the post excerpt in a hidden div in the email template so that email clients pick it up as the "preview" text when listing emails
- when no custom excerpt is provided the preview text is grabbed from post.excerpt which is the first 500 chars of the post.plaintext value
- post.plaintext formats links as "Link [http://url/]" which is unwanted in html email previews

- add a basic replacement to the post email serializer to remove any `[http://url/]` occurrences from the post excerpt before rendering the email content
2020-08-12 20:14:06 +01:00
Kevin Ansfield
163092f377 🐛 Fixed apostrophes not displaying correctly in Outlook for member emails
refs https://github.com/TryGhost/Ghost/issues/11536

- Outlook supports `&#39;` as a special char for apostrophes but not `&#apos;` which is what cheerio/juiced render
- adds a basic string placement to the email serializer to switch to the older style of special char
2020-08-12 20:01:42 +01:00
Kevin Ansfield
c7ff4c9e93 Moved email sending to the background job queue
no issue

- moves the meat of `pendingEmailHandler()` code into a new function `sendEmailJob()` that is passed over to the new job service
- lets the server keep processing email generation and sending when it receives a shutdown request rather than halting processing mid-send and ending up in a partial state
2020-08-12 17:02:14 +01:00
Fabien 'egg' O'Carroll
1294e3f92c
Replaced all usage of member models with members-api (#12117)
no-issue

* Added stripeSubscriptions relation to member model

This allows us to fetch the subscriptions for a member via standard
model usage, e.g. `withRelated: ['stripeSubscriptions']` rather than
offloading to loops and `decorateWithSubscriptions` functions, this is
more performant and less non-standard than the existing method.

* Updated serialize methods to match existing format

The current usage of `decorateWithSubscriptions` and the usage of
members throughout the codebase has a subscriptions array on a stripe
object on the member, this ensures that when we serialize members to
JSON that we are using the same format.

There is definitely room to change this in future, but this is an
attempt to create as few breaking changes as possible.

* Installed @tryghost/members-api@0.26.0

This includes the required API changes so that everywhere can use
members-api directly rather than models and/or helper methods
2020-08-12 14:17:44 +01:00
Kevin Ansfield
2efcf94645
Improved performance of sending newsletter emails (#12091)
no-issue

- switch from `membersService.api.members.list` to using bookshelf `Member.findPage()` with the `{paid: true}` filter to avoid per-member queries (N+1) to decorate members with subscriptions and a heavy post-fetch filter via `contentGating`
- add concurrency to the Mailgun API requests in `bulk-email` service to reduce overall time submitting API requests
- add debug statements with timing output for easier measurements
2020-08-06 15:19:39 +02:00
Gerardo Lopez Dueñas
275bf41ba3
Fixed bookmark card image size in fastmail (#11970)
closes #11907

The image in the bookmark card was being shown out of the bounds of
the card because of a general style `height: auto !important`.

I added a new `max-height` property to the image to avoid exceeding
parent height.
2020-08-03 18:38:45 +01:00
Hannah Wolfe
92446d85ea Changed member limit to be DRY & use raw query
- Member limit code was duplicated in 2 places unnecessarily
- Also used member api code that fetched members and subscriptions fully hyrated when we only need a count
- Using a raw query significantly improves performance here
2020-07-26 20:49:30 +01:00
Nazar Gargol
5f1060a8bf Renamed settings keys active_timezone to timezone
refs https://github.com/TryGhost/Ghost/issues/10318
refs 2614565d5a

- Renames to match referenced migration renames
- Fixed API responses so they are consistent with newly renamed fields
- Not returning lang and timezone keys from settings in API v2 ther rest should be returned in API v3/canary
2020-06-24 14:11:20 +12:00
Daniel Lockyer
5d30af1513 Fixed wrong require in post email serializer
refs https://github.com/TryGhost/Ghost/pull/11807

- this was originally reported in the PR above, but we could not
  reproduce it on master
- presumably the user had the latest version of moment installed for
  other purposes and so they were seeing the issue
- between moment 2.24.0 and 2.26.0, something must have changed which
  stopped the previous functionality working
2020-06-08 19:43:55 +01:00
Vikas Potluri
00c324fa4e
Moved core/server/lib/common/logging to core/shared/logging (#11857)
- Represents that logging is shared across all parts of Ghost at present
  * moved core/server/lib/common/logging to core/shared/logging
  * updated logging path for generic imports
  * updated migration and schema imports of logging
  * updated tests and index logging import
  * 🔥 removed logging from common module
  * fixed tests
2020-05-28 19:30:23 +01:00
Vikas Potluri
1bd8c18a16
Moved core/server/lib/url-utils to core/shared/url-utils (#11856)
* moved url-utils from server to shared
* updated imports of url-utils
2020-05-28 11:57:02 +01:00
Vikas Potluri
15d9a77092
Moved config from server to shared (#11850)
* moved `server/config` to `shared/config`
* updated config import paths in server to use shared
* updated config import paths in frontend to use shared
* updated config import paths in test to use shared
* updated config import paths in root to use shared
* trigger regression tests
* of course the rebase broke tests
2020-05-27 18:47:53 +01:00
Kevin Ansfield
ee20efc971 Fixed Outlook not centring content column in member emails
refs https://github.com/TryGhost/Ghost/issues/11756

- we removed the fixed `width: 600px` to fix gmail scaling but that means we need a new way of creating a 600px centre column in Outlook
2020-05-01 13:40:02 +01:00
Kevin Ansfield
28b24d2095 Fixed video embed card fallback display across gmail apps
refs https://github.com/TryGhost/Ghost/issues/11756

- bumped kg-default-cards package to generate fixed html for video embed fallbacks
2020-05-01 13:22:52 +01:00
Kevin Ansfield
2404710e66 🐛 Fixed tiny font sizes sometimes seen in member emails on gmail mobile apps
refs https://github.com/TryGhost/Ghost/issues/11756

- fixed gmail scaling problems
  - the `width: 600px` on `.container` was forcing gmail to always render at 600px wide and then use scaling to resize the email to fit the device width
  - for most emails gmail would also apply their own font resizing to compensate so it didn't look _too_ bad
  - some emails however would not trigger the font resizing, most notably when posts contained a feature image, which would result in very small text
  - removing the fixed `width: 600px` resolves the scaling problem and lets the email be truly responsive
- removed attribute selectors in the media query CSS
  - gmail does not support attribute selectors
  - attribute selectors used to be necessary for Yahoo Mail but this is no longer the case
- tested using litmus.com for all popular email clients
2020-05-01 11:06:52 +01:00
Hannah Wolfe
baa8118893 Refactor common pattern in service files
- Use array destructuring
- Use @tryghost/errors
- Part of the big move towards decoupling, this gives visibility on what's being used where
- Biting off manageable chunks / fixing bits of code I'm refactoring for other reasons
2020-04-30 20:48:42 +01:00
Kevin Ansfield
81bf1667fa Fixed video embed card fallback display in Outlook
refs https://github.com/TryGhost/Ghost/issues/11756

- updates `@tryghost/kg-default-cards` which includes a VML version of video embed card fallbacks
- fixes play button styling for Yahoo Mail
- adds a minimum height to video embeds so they appear more reasonable when images are not loaded
2020-04-30 12:08:06 +01:00
Kevin Ansfield
d255d98bbb Improved display of video embed cards in member emails
refs core/server/api/canary/oembed.js

- updated `kg-default-cards` to a version that will render a thumbnail and play button overlay for video embed cards when rendering to an email target
- added styling for video embed play button overlays to members email template
2020-04-29 16:44:20 +01:00
Kevin Ansfield
cd04262889 🐛 Fixed publish date in emails not respecting site's configured timezone
closes https://github.com/TryGhost/Ghost/issues/11659

- default `moment()` timezone is UTC and we store the `published_at` value in UTC
- fetch the configured timezone and convert the date into that timezone before formatting for inclusion in the email template
2020-04-27 11:38:11 +01:00
Kevin Ansfield
c5f7adf917 Fixed "view sent email" showing mailgun template variables
no issue

- the `email.{html,plaintext}` fields are only used to display what was sent in the email so it doesn't make sense to store the mailgun-specific content which can be confusing when viewing in the admin area
- store the raw serialized post content with a basic no-data replacement of replacement strings rather than the output of full data fetching and mailgun transformation
2020-04-20 15:35:33 +01:00
Kevin Ansfield
9981ea336c Changed {subscriber_firstname} replacement to {first_name}
no issue

- easier to remember and type
- removes need to remove the `subscriber_` prefix when accessing member properties
2020-04-20 14:25:58 +01:00
Kevin Ansfield
a09a6caf5f Fixed in-browser email preview showing raw replacement strings
no issue

- fixed plaintext templates being word wrapped and breaking across replacement strings
- updated `postEmailSerializer.serialize` to return the email template plus a replacements array that can be used for creating Mailgun-like recipient variable objects or more straight forward replacement
- updated email-preview API to work with the replacements data to show fallback data when previewing
2020-04-20 12:24:05 +01:00
Kevin Ansfield
e0e0a85a32 Updated test emails to fetch member data if email matches
no issue

- with the email replacements feature it's useful to have real member data when sending test emails from the PSM
- if the supplied email address matches a member then that member's data will be used for any replacements
2020-04-17 12:15:39 +01:00
Kevin Ansfield
d0393b6223 Fixed {subscriber_firstname} not picking up member's name in emails
no issue

- in our replacements the member properties are prefixed with `subscriber_` but this wasn't taken into account when requesting data from the member object
2020-04-17 12:15:39 +01:00
Kevin Ansfield
374b43ceab Fixed linting error 2020-04-17 11:32:29 +01:00
Kevin Ansfield
a801352c7f Added email card and replacement handling to member emails
no issue

- adjusted mega's post serializer to get full email contents
  - fetch `mobiledoc` from the API rather than the pre-rendered `html` and `plaintext`
  - re-generate `html` using the mobiledoc renderer with an "email" target so that the email-only card content is included
  - re-generate `plaintext` from the newly generated email html

- added replacement handling to mega's `getEmailData` function
  - find all of our `%%{replacement "fallback"}%%` instances in the html template and push them into a replacements array with the respective property on the member instance and desired fallback
  - transform the replacement for Mailgun compatibility. Mailgun uses `%recipient.variable_name%` for its template variables so we need to replace our custom replacement string with the compatible version. Our replacements system allows for the same replacement (`{subscriber_name}`) to be used multiple times and have different fallbacks, Mailgun doesn't support fallbacks so for each replacement we also need an indexed `variable_name` part so that we can put our fallbacks in the correct place
  - perform the same Mailgun template transformation for the plaintext version except we re-use the replacements array to avoid bloating the API request to Mailgun with duplicate template variables for every recipient
  - swapped `reduce` for a plain loop for easier readability
2020-04-17 10:23:03 +01:00
Rish
5f349b3ef7 🐛 Fixed missing publication icon in newsletter emails
no issue

Email template was incorrectly setting up publication icon url in case of subdirectory setup, leading to missing publication logo from newsletter emails in such cases. This adds the fix to use correct absolute url for publication icons in all setups.
2020-03-12 09:52:54 +05:30
Rishabh Garg
3815c0769a
🐛 Fixed incorrect email count on post publish (#11616)
no issue

The email data attached to a post when published with send email flag was not filtered on member access, and picked up the whole member list for email data. This resulted in incorrect data stored in emails table even in case of paid-members-only publish, and also incorrect count of "emails sent" being displayed on Admin.

NOTE: The actual emails being sent are still gated by member access, so no emails were sent to anyone without access, this only affected the associated email data and count. Also, the fix here will show correct email sent status for any future post, but will still show incorrect data for any already published posts as the email data in DB is already wrong and will probably need a migration
2020-02-24 16:34:07 +05:30
Rishabh Garg
9c1aa07ea8
Added host limit check for members email publish (#11534)
no issue
2020-02-13 10:43:36 +05:30
Daniel Lockyer
d76e76e1ef Fixed code linting issues
no issue

- just extraneous whitespace
2020-02-03 17:49:41 +00:00
Peter Zimon
a8b272cbd8 🐛 Fixed small image alignment for newsletters
no refs.
- fixed image alignment to be center for images smaller than newsletter content width
2020-01-21 13:21:45 +01:00
Peter Zimon
60c44d360b 🎨 Fixed test newsletter email subject
no issue.

- "[Test]" being appended (at the end of) the test email subject made it hard to scan for test emails. This fixes it by prepending "[Test]" to the subject.
2019-12-03 16:26:25 +01:00
Naz Gargol
201bef31f0 Added transaction support to pagination plugin (#11421)
Adds transaction support to `fetchPage` method. This is needed to be able to count members during the post publish transaction. 

This is the next iteration over initial quick-fix: 90905b0212

* Added transaction support to pagination plugin
    - This support is needed to be able to use `fetchPage` method in transactional context (example usecase was counting members when publishing post for emails)
* Passed transaction related options during email creation
    - Without this SQLite would hang in a transaction and eventually timeout
* Updated parameter name for consistency
2019-11-27 10:00:27 +00:00
Rish
628f9179dc Fixed URL import for post email serializer
no issue
2019-11-27 13:18:44 +05:30
Rish
9a53177544 Refactored unsubsribe url and getemailData methods
no issue
2019-11-27 10:58:32 +05:30
Rish
a3802c495d Fixed post serialization to keep plaintext in json
no issue

- Fixes missing plaintext on email preview
- Fixes tests
2019-11-27 10:58:32 +05:30
Nazar Gargol
63e6dd59fa Added missing await statement
no issue

- The 90905b0212 refactor missed the statment which is causing email to not being sent
2019-11-27 09:39:48 +07:00
Kevin Ansfield
90905b0212 Fixed emails sending when scheduled post is published
no issue

- the schedules controller wraps the post creation in a transaction
- we need to pass that transaction through to all other queries, especially on sqlite where a non-transaction query inside a transaction will lock up because there's only 1 connection available
- updates our model method calls to pass through the transaction options
- switches the members service `list()` call to a direct model `findAll()` call to avoid going through our pagination plugin because the raw knex query does not respect the transacting option
2019-11-26 17:43:29 +00:00
Rish
b9dd0d2b94 Refactored email handling to be consistent for test and newsletter emails
no issue
2019-11-26 21:41:01 +05:30
Nazar Gargol
9ff5fecbaf Fixed knex connection pool errors when scheduling a posts
no issue

- A subquery in mege service that creates email record wasn't using 'options' object needed to track transactions
2019-11-26 17:44:42 +07:00
Peter Zimon
4790e64256 Updated unsubscribe copy 2019-11-26 11:03:14 +01:00
Rish
7209abb729 Updated unsubscribe url for preview email
no issue
2019-11-26 15:14:52 +05:30
Rish
e6f74c63db Fixed post serialization for test emails
no issue
2019-11-26 11:59:41 +05:30
Peter Zimon
d7d5d9a13d Refined email styles 2019-11-22 11:40:56 +01:00
Peter Zimon
e82f706afa Email design refinements 2019-11-20 18:27:52 +01:00
Naz Gargol
c2aec69af9
Added email retry logic for failed batches (#11402)
no issue

- When whole email batch fails we want to allow retrying sending a batch when post is republished
- Refactored naming for email event handling in mega
2019-11-18 21:28:54 +07:00
Naz Gargol
c99f40957e
Improved mega error handling (#11393)
no issue

- Increased default mailgun retry limit to 5
- Handling retry logic closer to SDK layer gives less future manual handling
- Allowed failing request to be passed through to the caller
- To be able to handle failed requests more gracefully in the future we need all available error information to be given to the caller
- The previous method with `Promise.all` would have rejected a whole batch without providing details on each specific batch.
- Limited data returned with a failed message to batch values
- Added better error handling on mega layer
- Added new column to store failed batch info
- Added reference to mailgan error docs
- Refactored batch emailer to respond with instances of an object
- It's hard to reason about the response type of bulk mailer when multiple object types can be returned
- This gives more clarity and ability to check with `instanceof` check
2019-11-15 18:25:33 +07:00
Kevin Ansfield
ee47dd4dae
Added /unsubscribe/ route to the front-end (#11394)
no issue

- adds new router to the frontend for handling unsubscribe
- default template lives in `core/server/frontend/views/unsubscribe.hbs`
- `{{error}}` is present and contains the error message when unsubscribe fails
- `{{member}}` is present and contains the member email
- updated unsubscribe url to match the new format
2019-11-15 09:36:49 +00:00
Rish
52eb3ca9da Converted all link tags in preview to open in new tab
no issue
2019-11-15 11:26:43 +05:30
Rish
6d1de1b912 Fixed post not showing authors in email
no issue
2019-11-14 10:45:05 +05:30
Rish
b7df5c360d Updated send test email to use common email data 2019-11-13 22:53:33 +05:30