Commit Graph

4388 Commits

Author SHA1 Message Date
Kevin Ansfield
76c1b60a4d
Added schema+migration for email_{batches,recipients} tables (#12192)
no issue

We want to store a list of recipients for each bulk email so that we have a consistent set of data that background processing/sending jobs can work from without worrying about moving large data sets around or member data changing mid-send.

- `email_batches` table acts as a join table with status for email<->email_recipient
  - stores a provider-specific ID that we get back when submitting a batch for sending to the bulk email provider
  - `status` allows for batch-specific status updates and picking up where we left off when submitting batches if needed
  - explicitly tying a list of email recipients to a batch allows for partial retries
- `email_recipients` table acts as a join table for email<->member
  - `member_id` does not have a foreign key constraint because members can be deleted but does have an index so that we can efficiently query which emails a member has received
  - stores static copies of the member info present at the time of sending an email for consistency in background jobs and auditing/historical data
2020-09-14 12:21:58 +01:00
naz
cbdc91ce48
Added Location header to API's POST request responses (#12186)
refs #2635

- Adds 'Location' header to endpoints which create new resources and have corresponding `GET` endpoint as speced in JSON API - https://jsonapi.org/format/#crud-creating-responses-201. Specifically:
    /posts/
    /pages/
    /integrations/
    /tags/
    /members/
    /labels/
    /notifications/
    /invites/

- Adding the header should allow for better resource discoverability and improved logging readability
- Added `url` property to the frame constructor. Data in `url` should give enough information  to later build up the `Location` header URL for created resource.
- Added Location header to headers handler. The Location value is built up from a combination of request URL and the id that is present in the response for the resource. The header is automatically added to requests coming to `add` controller methods which return `id` property in the frame result
- Excluded Webhooks API  as there is no "GET" endpoint available to fetch the resource
2020-09-14 22:33:37 +12:00
Talha
afa976066a
🐛 Fixed table constraint error when updating member's email with an already existing email (#12178)
closes #12045

- When member's email is updated to an already existing email of different member it caused table's unique constraint error, which was not handled properly. 
- Added handling for this error similar to one in members `add` method.
2020-09-10 16:03:57 +12:00
Nazar Gargol
5582d030e3 Added routes.yaml content checksum storage to the db
closes #11999

- When the routes.yaml file changes (manually or through API) we need
to store a checksum to be able to optimize routes reloads in the future
- Added mechanism to detect differences between stored and current routes.yaml hash value
- Added routes.yaml sync on server boot
- Added routes.yaml handling in controllers
- Added routes hash synchronization method in core settings. It lives in core settings
as it needs access to model layer. To avoid coupling with the frontend settings it accepts
a function which has to resolve to a routes hash
- Added note about settings validation side-effect. It mutates input!
- Added async check for currently loaded routes hash
- Extended frontend settings loader with async loader. The default behavior of the loader is
to load settings syncronously for reasons spelled in 0ac19dcf84
To avoid blocking the eventloop added async loading method
- Refactored frontend setting loader for reusability of  settings file path
- Added integrity check test for routes.yaml file
2020-09-10 10:54:57 +12:00
Talha
c8a6939b2e
🐛 Fixed error caused by accepting invitation with existing email (#12172)
closes #12060

- A 500 error what happening when invited user provided an email that is associated with an existing user
- Additional validation for existing email address was added to prevent invalid data hitting db constraint error
2020-09-09 11:58:53 +12:00
Rish
3c7a95aefa Added members support address to site data
refs https://github.com/TryGhost/members.js/issues/90

- Includes new members_support_address in member site data endpoint for Portal
2020-09-08 20:48:57 +05:30
naz
fcc6401445
Added routes_hash setting (#12171)
refs #11999

- The `routes_hash` setting will be used during the boot process to update the hash
of currently loaded routes.yaml file in case it's different from last restart
2020-09-07 21:54:55 +12:00
Nazar Gargol
4606c93e4f Refactored headers function to use async/await
no issue

- The async/await syntax makes it easier to reason about the code. Because adding 'Location' header is in the works it's a prep-work in a sense
2020-09-07 15:36:06 +12:00
Rish
2d2fa1a0ba Fixed missing subscription data in member update endpoint
closes https://github.com/TryGhost/members.js/issues/94

- The member-api package was recently updated to work directly with models and needs explicit `withRelated` options to attack relations
- Without options, the endpoint was returning the default member data without subscriptions attached, which in Portal showed paid member as free
- Fix updates the middleware for updating member data to correctly pass the relations needed to populate the member
2020-09-04 17:11:04 +05:30
Rish
dd6ac57aca Fixed missing domain for default support address
no issue

- By default for new sites, support address is set same as from address to `noreply` , with full email address using the domain for `@`
- For newsletter emails, the support address was missing the default site domain to be added to address if its `noreply`
- Fix updates the support address to use the same format as from address and add relevant domain for default case
2020-09-03 16:34:47 +05:30
Kevin Ansfield
936d7a315a 🐛 Fixed incorrect status used for trial subscription query
refs 96c18edc82

- replaced `trialed` with `trialing` to correctly match the status used in Stripe
2020-09-02 13:07:48 +01:00
Rish
696e60dd51 🐛 Fixed missing member email on unsubscribe page
no refs

- The `update` method in members-api package was edited to return Model object instead of JSON directly - TryGhost/Members@a28bcc5
- This unsubscribe handler was returning the raw member object returned from `update` method, which is now a model object and not able to access `member.email`
- Fix updates the unsubscribe request handler to return the member JSON again
2020-09-02 12:03:10 +05:30
Kevin Ansfield
96c18edc82 🐛 Fixed members with Stripe trials not receiving paid-members newsletters
no issue

- members who have trial subscriptions added directly via Stripe will have a status of `"trialed"` in their Ghost subscription
- the `paid: true` filter was not taking that into account meaning trial users were not receiving newsletters sent to paid members even though they have a "paid" subscription
2020-09-02 00:34:26 +01:00
Rish
806d0f8e21 Updated sender address for ownership verification emails to new address
no issue

- We used existing "from" address as sender for mails sent to new email address for verification, but that breaks the flow to update if the current "from" address has DMARC policy set.
- This updates the flow to always send the ownership verification email TO the new address and FROM the new address which both verifies the email deliverability for new address and ownership
2020-08-31 18:09:38 +05:30
Rish
8d022ecfb5 Updated newsletter emails to include reply-to address
no issue

- The newsletter emails are sent out with `from` address as sender
- The new `members_reply_address` setting is now used to set reply-to address for emails, which can be either newsletter or support address
2020-08-31 18:09:38 +05:30
Rish
ca8b77182c Updated sender address for member auth emails to support address
no issue

- Member auth emails were previously using the `from` address as sender
- New `members_support_address` was introduced with default as original "from" address
- Auth emails use the new support address as sender
2020-08-31 18:09:38 +05:30
Rish
cbb3664bdd Added handling for new members support/reply email addresses
no issue

- Updated magic link generation and validation methods for email update API to handle new support address
- Updated importer to ignore the new support address as it can only be updated via verification
- Updated members service to listen on settings edit for new support/reply address fields as well
- Updated tests to include the new settings
2020-08-31 18:09:38 +05:30
Rishabh Garg
bc2f6eb8e9 Added migrations for support and reply email address setting (#12163)
no issue

- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account, public setting
- `members_reply_address` - Where you receive responses to newsletters
2020-08-31 17:16:53 +05:30
Rish
8cf9dc05e3 Revert "Added migrations for support and reply email address setting (#12163)"
This reverts commit 2216019a86.
2020-08-31 14:25:29 +05:30
Nazar Gargol
eb8c0fcff9 🐛 Fixed Tags API v2 to return posts count
closes #12167

- Tags API v2 was ignoring `count.posts` include parameter.
- Regression was introduced with a3f693b472
- Introduced regression tests across all Content API versions to avoid similar bug in the future
2020-08-31 18:46:35 +12:00
Rishabh Garg
2216019a86
Added migrations for support and reply email address setting (#12163)
no issue

- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account
- `members_reply_address` - Where you receive responses to newsletters
2020-08-27 20:27:50 +05:30
Nazar Gargol
e8248cdc9e Refactored members imporeter in preparation for jobs
no issue

- This refactor extracts labels related code into a separate module for easier reuse by the "job-aware" batched importer
2020-08-27 21:47:00 +12:00
Nazar Gargol
262d33b73c Improved import_label creation logic
no issue

- When an import was done and there were no "global labels" present Ghost created generic `import-[data]` label which later helped to find a specific batch of imported data
- It did not make sense to create such generic label when user provided their own unique label
- The rules that work now are:
1. When there is no global provided Ghost generates on and removes it in case there are no imported records
2. When there is a unique new global label provided no new label is generated, but the label stays even if there are no imported records
2020-08-27 19:07:27 +12:00
Kevin Ansfield
f30d3cd2c2 Switched to bookshelf count in getTotalMembers()
no issue

- tested performance between knex raw, knex `count()` and bookshelf `count()` and found no difference over 1000 iterations of each (each ~19,500ms +- 500ms for 104k members locally)
- switched to using bookshelf as the code is the simplest
2020-08-27 01:52:36 +01:00
Kevin Ansfield
a995e9cc89 Allowed custom and search queries to be used with .findAll()
no issue

- extract filtering of an collection into a separate function
- use extracted function in `findAll()` so that it's query behaviour matches `findPage()`
2020-08-27 01:09:07 +01:00
Kevin Ansfield
5e64f113d5 Skipped separate count query in .findPage() for limit="all" requests
no issue

- for large result sets or complex queries the count query itself can be quite time consuming
- when `limit: 'all'` is passed as an option there's no need to perform a separate count query because we can determine the pagination data from the final result set
- skipped count query when `limit: 'all'` option is present
- re-ordered comments to be closer to the code they reference (ie, why we have our own count query instead of Bookshelf's `.count()`
2020-08-27 01:09:07 +01:00
Nazar Gargol
9fe5007929 Removed date handling validation in members importer logic
no issue

- This is handled on input sanitization layer with date
format check in JSON schema validation, so there's no need to do this
check again in the importer.
2020-08-26 17:25:24 +12:00
Nazar Gargol
544382b651 Updated boolean values validations for members uploads
no issue

- When the value is part of `enum` the `null` value is no longer
recognized by `null` type in JSON schema validator.
2020-08-26 17:23:36 +12:00
Nazar Gargol
95044e3ba0 Added error handling for failed member imports
no issue

- When bulk insert fails there is no transactional logic to revert
related records form being inserted. Also, previously there were no
attempts to "retry" the insert.
- To avoid complex retry logic, an iterative one-by-one insert retry
approach was taken. If this becomes a bottleneck in the future, the
retry algorithm could be improved.
- To avoid a lot of code duplication refactored model's `bulkAdd` & `bulkDestroy`
methods to use 'bulk-operations' module.
- Updated error handling and logging for bulk delete operations. It's very
unlikely for error to happen here,  but still need to make sure there is
a proper logging in place to trace back the failure.
- Added debug logs. This should improve debugging experience and
performance measurements.
- Added handling for unrecognized errors. Handling inspired by current unrecognized
error handling by ghost importer -10e5d5f3d4/core/server/data/importer/importers/data/base.js (L148-L154)
2020-08-26 17:11:35 +12:00
Nazar Gargol
45bca2b104 Fixed linting for await statements
refs 9d65de4b10
2020-08-25 19:30:02 +12:00
Nazar Gargol
9d65de4b10 Removed created import labels when no records were imported
no issue

- When no members are succesfully imported through CSV import process the import label should not be created. Otherwise after multiple failed attempts to import there are orphaned labels in the system
2020-08-25 19:23:05 +12:00
Kevin Ansfield
e07e0a17c6 🐛 Fixed bookmark card display in member emails when using Gmail+Chrome
no issue

- updates `@tryghost/kg-default-cards` which contains two fixes
  - removes email-specific output being added to post html (had no visual impact due to use of conditional comments but keeps rendered html smaller+cleaner)
  - adds a background-url style to the thumbnail container to give two options for styling
- updates member email template styling to hide the `<img>` element in bookmark cards and use a background image instead to get consistent rendering across email clients
2020-08-24 18:21:09 +01:00
Peter Zimon
75f5ab637d Added minor member newsletter customisations via config
no issue
- adds a `members:emailTemplate` config object
  - `showSiteHeader` - defaults to `true`, shows the site title and icon in member emails
  - `showPoweredBy` - defaults to `false`, adds a "Publish with Ghost" button to member email footer
- updates member newsletter email template with hideable site header and "powered by" badge
2020-08-24 10:44:22 +01:00
Rish
297b63edce 🐛 Fixed response for members API update endpoint
closes https://github.com/TryGhost/members.js/issues/87

- The `update` method in members-api package was edited to return Model object instead of JSON directly [here](a28bcc5b2a)
- This caused the update member API on member endpoint to return partial response only as most properties couldn't be fetched
- Fix updates the middleware to correctly call `toJSON` before formatting response
2020-08-24 13:38:21 +05:30
Nazar Gargol
91d1417df2 Fixed email newsletter error when post has no content
closes #12158
2020-08-24 19:46:46 +12:00
Daniel Lockyer
0ab190e83c Merged 3.30.2 into master
* 3.30.2:
  Bumped @tryghost/members-api to 0.28.1 in lockfile
  Bumbed @tryghost/members-api to 0.28.1
  🐛 Fixed unable to delete member when stripe is connected
2020-08-21 12:06:01 +01:00
Rish
87e8298ca7 🐛 Fixed unable to delete member when stripe is connected
closes https://github.com/TryGhost/Ghost/issues/12150

- The `destroy` method on members-api method was called incorrectly with not enough params
- It expects both `data` and `options` passed down [here](https://github.com/TryGhost/Members/blob/master/packages/members-api/lib/users.js#L14)
- Missing `options` in method call throws error as we read `cancelStripeSubscriptions` property on options if stripe is connected
- Fix passes both data and options to the destroy method
2020-08-21 11:13:07 +01:00
Rishabh Garg
c7f3837c54
Added admin endpoint for editing member subscription (#12145)
refs #12127

- Adds new `editSubscription` endpoint for members admin API which allows updating individual subscription for a member - `PUT /members/:id/subscriptions/:subscription_id/`
- `editSubscription` has same permissions as member's `edit` endpoint
- Currently allows toggling of cancellation at period end for an active subscription
2020-08-20 17:28:11 +05:30
Nazar Gargol
5670d99f46 Improved error handling for batch deleted records
no issue

- Similar handling to one introduced in 8418c829de
- Having granular tracking for failed to remove id's would make it possible to return more specific errors to the client
2020-08-20 19:11:41 +12:00
Nazar Gargol
8418c829de Improved error handling for batch inserted records
no issue

- Similar to 3a594ce22e this adds error handling to batch operations done outside models
2020-08-20 19:01:17 +12:00
Nazar Gargol
f8adb4a672 Refactored bulk insert/delete operations into separate module
no issue

- Moved bulk db operations outside of importer module to create clearer separation of responsibilities
2020-08-20 18:42:38 +12:00
naz
3a594ce22e
Improved error handling for batch inserted member records (#12146)
no issue

- When batch insert fails handling should be more granular and aim to retry and insert as many records from the batch as possible.  
- Added retry logic for failed member's batch inserts. It's a sequential insert for each record in the batch. This implementation was chosen to keep it as simple as possible
- Added filtering of "toCreate" records when member fails to insert. We should not try inserting related members_labels/members_stripe_customers/members_stripe_customer_subscriptions records because they would definitely fail insertion without associated member record
2020-08-20 17:41:47 +12:00
Nazar Gargol
2e769e3122 Fixed handling for Stripe connected members import
no issue

- When stripe is disconnected and there are Stripe-connected records present in imported set they should not be processed and proper error should be thrown
2020-08-20 17:08:19 +12:00
Nazar Gargol
4498b4624a Fixed label serialization in members bulk importer
no issue

- After a refactor logic was missing trimming logic and handling for empty labels
2020-08-20 16:43:42 +12:00
Nazar Gargol
32fd913ac3 Fixed default subscribed value for member model in the importer
no issue

- The default value should not be a string but rather a boolean
2020-08-19 23:42:48 +12:00
Kevin Ansfield
d8e319af88 🐛 Fixed card spacing and caption styling for member emails in Outlook
closes https://github.com/TryGhost/Ghost/issues/12139

- once the email content has been rendered in the post serializer, perform some whole-content transformation of `figure` and `figcaption` to `div` using cheerio
- juiced will have already inlined the elements styles so there's no need to adjust the template's stylesheet
2020-08-19 12:34:14 +01:00
Nazar Gargol
616767d0ec Fixed members bulk import's createdBy field assignment
no issue

- Method signature has been changed with cada88975a, the call needed an update as well
2020-08-19 17:14:45 +12:00
Jeremy Davidson
681c96b5e9
🐛 Fixed bookmark card author/publisher spacing in member emails (#12134)
closes #12078

- Root cause was that pseudo class .kg-bookmark-author:after was not getting inlined to email newsletter or its preview.
  - That is where the margin and bullet-point content are added between author and publisher.
- Dependency juice supports an option inlinePseudoElements which is false by default.
- Fix was to set inlinePseudoElements to true when serializing post email.

Co-authored-by: Jeremy Davidson <jeremy@crossingcontour.com>
2020-08-18 16:18:44 +01:00
Hannah Wolfe
9f291b2d10 Improved testmode logging of shutdown status
- testmode allows us to test Ghost's behaviour with long running requests and jobs
- moved all the testmode logging to only happen if Ghost starts successfully to make it clearer what is happening
- added a (currently very dirty) bit of code to fetch the jobService and output the queue status giving us a similar
     output for both open connections and jobs
- all of this allows for debugging exactly what Ghost is doing whilst it is processing a shutdown request
2020-08-18 15:48:42 +01:00
Kevin Ansfield
cefcdad090 🐛 Fixed emails appearing very wide in Outlook and improved email image sizes
closes https://github.com/TryGhost/Ghost/issues/11536

- bumps `@tryghost/kg-default-cards`
  - image and gallery cards now output `width/height` attributes on `img` elements with a max width of 600px
  - uses resized images where possible to keep email weight down
- adds `height: auto` style to image card images so that the `height` attribute does not cause distortion at smaller screen widths
2020-08-18 15:33:54 +01:00