Commit Graph

8415 Commits

Author SHA1 Message Date
Thibaut Patel
bda4e0da70 Fixed Personal Tokens migration file name
no ref
2020-11-20 10:35:31 +01:00
Thibaut Patel
5fefa9fe96 Disabled Personal Tokens of inactive/locked users
no issue
2020-11-20 09:53:18 +01:00
Thibaut Patel
d48febbbd8 Added Personal Token removal when a user is deleted
no issue
2020-11-20 09:53:18 +01:00
Thibaut Patel
4edccfd2f7 Added the Admin API and tests for Personal Tokens.
no issue
2020-11-20 09:53:18 +01:00
Thibaut Patel
be4146e324 Enabled authentication using an api key with a user_id
no issue

- Enables the Personal Tokens feature.
2020-11-20 09:53:18 +01:00
Thibaut Patel
214ed405cc Added the user_id field to the api_keys table
no issue

- Migration related to the Personal Tokens feature.
2020-11-20 09:53:18 +01:00
Fabien 'egg' O'Carroll
38d19c999d
Removed search params from custom signup redirects (#12386)
refs #12366

This means that custom redirects will not trigger the built in notifications handled by Portal.
2020-11-19 14:47:08 +00:00
Daniel Lockyer
49197a6597 Merged 3.38.2 into master
* origin/3.38.2:
  v3.38.2
  Updated Ghost-Admin to v3.38.2
  🐛 Added missing Bluebird require in v2 API
2020-11-19 13:43:18 +00:00
Daniel Lockyer
20bd729191 Updated Ghost-Admin to v3.38.2 2020-11-19 13:40:08 +00:00
Daniel Lockyer
c72ecf6a96 🐛 Added missing Bluebird require in v2 API
- this was missed in the original pass when removing the global Promise
  override
2020-11-19 13:36:15 +00:00
Fabien 'egg' O'Carroll
9ba3e96790
Redirected to custom path on successful signup (#12372)
refs #12366 

This implements redirection based on the settings for successful member sign up!
- Removes support for redirecting to `req.path` afterwards, this was never used and
  we now have a more configurable implementation.
- Retains redirection to the homepage for unsuccessful sign up (invalid/expired token)
2020-11-19 09:58:32 +00:00
Fabien 'egg' O'Carroll
2600a20f54
Added new settings for members signup redirects (#12370)
refs #12366

Adds two new settings for redirecting members after signup, based on if
they're paid or free, defaulting to the current redirect of "/"
2020-11-17 17:17:54 +00:00
Daniel Lockyer
dccbaf847c Updated Ghost-Admin to v3.38.1 2020-11-17 16:42:06 +00:00
Fabien 'egg' O'Carroll
8ad995203e
🐛 Fixed migration for SQLite3 (#12371)
no-issue

The current version of SQLite3 bundled with Ghost supports a maximum of 999
variables, we use one variable for the SET value and so we're left with 998 for the
WHERE IN clause values.

refs: https://forum.ghost.org/t/unable-to-start-ghost-3-38-0-locally/18322
2020-11-17 16:39:42 +00:00
Daniel Lockyer
99c24ca109 Updated Ghost-Admin to v3.38.0 2020-11-17 13:53:34 +00:00
Fabien 'egg' O'Carroll
bfc152bede
Fixed handling of email_recipient_filter option (#12369)
no-issue

This logic would assume that the option was always passed at the point
of publishing the post, which is not the case for scheduled posts.

Instead of setting the property to 'none' when the option is not
present, we take the approach of ONLY setting the propery when
1. It is present and not 'none'
2. The post is being published or scheduled

This means that scheduled posts will have the property set correctly,
and any future publishing will leave the it in the original state
2020-11-17 11:00:03 +00:00
Rish
f53ab0f52b Updated default config for newsletter badge
no refs

- Migrations were using previous default for badge which was set to `false`
- Default config for badge was `false` which caused migration to switch off the badge, updated to `true`
2020-11-17 16:06:31 +05:30
fueko
8db835a057
Removed unneeded "type='text/css'" in members styles (#12329)
no issue

- Removed unneeded "type='text/css'" attribute from style tag when loading member related styles. 
- Having this attribute makes W3C validation fail.
- Reference on style tag's "type" attribute - https://developer.mozilla.org/en-US/docs/Web/HTML/Element/style#attr-type
2020-11-17 15:28:51 +13:00
Naz
6268d9c324 Refactored UrlGenerator simplifying conditional logic
no issue

- The logic inside the if/esle conditions was duplicated and made it confusing to reason about.
- This change is not effecting the underlying logic at all
2020-11-17 13:09:22 +13:00
Naz
05e12d4b5a Added jsdocs to UrlGenerator/Resource classes
no issue

- The @param {Resource} syntax was not semantically correct. After the change TypeScript is picking up a correct parameter
2020-11-17 13:05:40 +13:00
Daniel Lockyer
33bc3c9011
Refactored column migrations to use helpers (#12360)
- these helpers remove a lot of the duplicated code that we had when doing up/down column migrations, and provides a much shorter way of doing this in the future
2020-11-16 11:55:51 +00:00
Peter Zimon
02d78a89aa Udpated email template footer spacings
no refs.
- updated padding between Ghost badge and rest of the footer
2020-11-16 12:31:37 +01:00
Rish
2d04bfbac9 Enabled newsletter badge by default
no issue

Updated newsletter badge to be shown by default which was previously set to be hidden. Users can still control the visibility via Email settings in Ghost-Admin
2020-11-16 15:55:05 +05:30
Peter Zimon
cfac18cb32 Newsletter email template refinements
no refs.
- updated color styles, links for footer text
- refined spacings for email footer paragraphs
2020-11-13 16:58:56 +01:00
Fabien 'egg' O'Carroll
3f9053ae94
Added setting for newsletter footer content (#12359)
no-issue

We want to give users to ability to customise the content of their newsletter, and the first step
toward that is a setting in which we can store text or html to embed in the template
2020-11-13 13:18:47 +00:00
Fabien 'egg' O'Carroll
4604ba1587
Fixed backward compatibility for send_email_when_published (#12357)
no-issue

* Handled send_email_when_published in Posts API

This restores backwards compatibility of the Posts API allowing existing
clients to continue to use the `send_email_when_published` flag. This
change uses two edits, which is unfortunate. The reason being is that
this is an API compatibility issue, not a model issue, so we shouldn't
introduce code to the model layer to handle it. The visibility property
of the model is used to determine how to fall back, and because it can
be left out of the API request, and relies on a default in the settings,
we require that the model decide on the `visibility` before we run our
fallback logic (or we duplicate the `visibility` default at the cost of
maintenance in the future)

* Dropped send_email_when_published column from posts

Since this column is not used any more, we can drop it from the table.
We include an extra migration to repopulate the column in the event of
a rollback

* Updated importer to handle send_email_when_published

Because we currently export this value from Ghost, we should correctly
import it. This follows the same logic as the migrations for this value.

* Included send_email_when_published in API response

As our v3 API documentation includes `send_email_when_published` we must
retain backward compatibility by calculating the property.

* Fixed fields filter with send_email_when_published

* Added safety checks to frame properties

Some parts of the code pass a manually created "frame" which is missing
lots of properties, so we check for the existence of all of them before
using them.

* Fixed 3.1 migration to include columnDefinition

We require that migrations have all the information they need contained
within them as they run in an unknown state of the codebase, which could
be from the commit they are introduced, to any future commit. In this
case the column definition is removed from the schema in 3.38 and the
migration would fail when run in this version or later.
2020-11-11 13:03:41 +00:00
Peter Zimon
215bfd0a7a
Added newsletter design settings (#12352)
refs https://github.com/TryGhost/Ghost/issues/12355

- Adds new default settings for newsletter customisations - `newsletter_show_badge`, `newsletter_show_header` and `newsletter_body_font_category`
- Adds migrations to update group for new settings
- Add migration to update settings based on existing config value for newsletter settings
- Passes new newsletter settings to newsletter template and updates design based on them
- Fix tests
2020-11-11 18:26:11 +05:30
Daniel Lockyer
f568c4a65e Moved Sentry noop function into block
- this function can be reduced in scope as it's only used in one of the
  branches
2020-11-10 11:04:21 +00:00
Naz
5b6fe31002 Added support for jobs defined as modules
no issue

- These changes should allow easy testing of scheduled jobs and provide an implementation example for future jobs within Ghost
- Added experimental support to run jobs based on functions defined in modules
- Extendeded testmode "schedule" endpoint with an optional job name parameter which picks up job from "jobs" folder
- Adjusted scheduleJob method use and passed "schedule" as first parameter
- Bumped job-manager version to allow for all new functionality
2020-11-10 17:29:35 +13:00
Naz
d6505899a7 Moved testmode routes into their own folder
no issue

- Living in separate folder will allow adding more testmode specific modules (for example test jobs)
- Mimicks the folder structure of how v2/canary routes are structured better
2020-11-10 15:01:28 +13:00
Fabien 'egg' O'Carroll
6140a98351
Updated newsletter functionality to use email_recipient_filter (#12343)
no-issue

* Used email_recipient_filter in MEGA

This officially decouples the newsletter recipients from the post
visibility allowing us to send emails to free members only

* Supported enum for send_email_when_published in model

This allows us to migrate from the previously used boolean to an enum
when we eventually rename the email_recipient_filter column to
send_email_when_published

* Updated the posts API to handle email_recipient_filter

We now no longer rely on the send_email_when_published property to send
newsletters, meaning we can remove the column and start cleaning up the
new columns name

* Handled draft status changes when emails not sent

We want to reset any concept of sending an email when a post is
transition to the draft status, if and only if, and email has not
already been sent. If an email has been sent, we should leave the email
related fields as they were.

* Removed send_email_when_published from add method

This is not supported at the model layer

* Removed email_recipient_filter from v2&Content API

This should not be exposed on previous api versions, or publicly

* Removed reference to send_email_when_published

This allows us to move completely to the email_recipient_filter
property, keeping the code clean and allowing us to delete the
send_email_when_published column in the database. We plan to then
migrate _back_ to the send_email_when_published name at both the
database and api level.
2020-11-06 17:32:23 +00:00
Fabien O'Carroll
44b43a4755 Fixed error with removal of global.Promise override
refs #12182

These were missed in the cleanup and were causing errors.
2020-11-06 16:49:10 +00:00
Fabien O'Carroll
0ce8e3171b Bound SSOAdapter methods to their instance
no-issue

This allows custom SSO adapters to store instance config and refer to
in their methods
2020-11-05 13:08:11 +00:00
Kevin Ansfield
aface9ed4c Enabled Mailgun open tracking when dev experiments is enabled
no issue

- set `emails.track_opens` to `true` when the `enableDeveloperExperiments` flag is set
- update mailgun bulk-email provider to pass the open-tracking header to Mailgun when the email's `track_opens` flag is set
2020-11-05 12:55:08 +00:00
Daniel Lockyer
c8f23873c6 Fixed wording on warning message
- derp
2020-11-05 12:18:28 +00:00
Daniel Lockyer
d8fdf2052c Added check for missing DB data
- this check should never really be hit unless the DB has missing data,
  but the code tries to read from an undefined variable and crashes, which we
  can protect against
- this can be triggered by doing a `knex-migrator init` followed by a
  `knex-migrator rollback --v 3.0 --force`
2020-11-05 12:13:58 +00:00
Daniel Lockyer
a3c8b032de Added missing Bluebird requires
- 504509bb6 removed the global override for Promise
- there are a bunch of places in code that use Bluebird Promise methods,
  but Bluebird wasn't being imported in these places
- this would have thrown errors all over the place
2020-11-05 11:55:14 +00:00
Kevin Ansfield
d41c2f8446
Added migration for emails.track_opens column (#12331)
no issue

- tracking of bulk email opens can be enabled/disabled over time, if we're calculating analytics for emails we don't want emails which didn't have tracking enabled skewing the results so we need a record of whether tracking was enabled for each email
2020-11-05 11:39:53 +00:00
Naz
c7541a1fe0 Added scheduled job support in testmode
no issue

- We need an ability to early test scheduled jobs to refine the new feature and it's API. Should be used with caution
- To schedule an example scheduled job every 30 seconds run following request: `curl http://localhost:2368/ghost/api/schedule/every%2030%20seconds`
2020-11-05 17:46:57 +13:00
Naz
fae05558f3 Added JSDoc comments to redirects service
refs #11085
2020-11-05 12:45:34 +13:00
Naz
f69526c140 Fixed error logged for absent redirects config
refs #11085

- Incorrect usage error was logged to the output when there was no recirecst configuration file present in the system. Previously an empty string was returned in such situation, resulting in "ENOENT" error, which was ignored through special handling.
- The fix resembles logic in redirects async getter function where empty array is returned when the config file does not exits.
- Attempting to read unexistent config should not ever happen and will be handled on the config service layer, this is why special "ENOENT" handling has been removed
2020-11-05 12:45:02 +13:00
Fabien O'Carroll
601b6ce174 Added migration to populate recipient_filter
no-issue

This populates the recipient_filter column for existing emails
based on the visibility of the related post at the time of the
migration.
2020-11-04 16:29:54 +00:00
Fabien O'Carroll
918c721bd1 Added recipient_filter column to emails table
no-issue

This column will allow us to store the canonical recipient filter on the
email resource giving us a detailed log of which members an email was
intended for
2020-11-04 16:29:54 +00:00
Fabien O'Carroll
dd0f80c9fa Added migration to populate email_recipient_filter
no-issue

This populates the email_recipient_filter column for existing posts
based on 1. if they were sent as an email and 2. their visibility.
2020-11-04 16:29:54 +00:00
Fabien O'Carroll
3b0a84b33b Added email_recipient_filter column to posts table
no-issue

This column will allow us to decouple the recipients of newsletter from
the `visibility` of a post, allowing us to send emails to specifically
free members, or to send paid posts as newsletters to all members.
2020-11-04 16:29:54 +00:00
Daniel Lockyer
9b45ea6f7d Added original error to logged error message
- without this, the original error stacktrace is hidden which makes
  issues harder to track down
2020-11-04 14:00:59 +00:00
Kukhyeon Heo
504509bb67
Removed global.Promise override (#12182)
closed #11943 

* Remove global.Promise
* Fix brute-knex bluebird error.
* Fix api-acceptance tests.
* Fix unit tests
2020-11-04 10:55:47 +00:00
Naz
e3e15646eb Updated redirect error message
refs #11085

- The logic which uses this error deals with an input yaml file thus the change.
- Related discussion here - https://github.com/TryGhost/Ghost/pull/12187#discussion_r508144220
2020-11-04 12:26:22 +13:00
Kukhyeon Heo
5715aa2155
Added .yaml format support in redirects configuration (#12187)
closes #11085

- Ghost has been using YAML format for other configurations (e.g. routes). The plan is to move to this format for all user-edited settings files. By default JSON format is still used in Ghost Admin API v2/v3, but will be changed to YAML in API v4. Check referenced issue for more context.
- New format supports all the features available before. The main noticeable change is the structure of config file. It is now grouped by redirect HTTP code instead of specifying `"permanent": true | false` attribute for each config property. Example format for YAML config:
```
302:
  /from-url/: /to-url/

301:
  /category/([a-z0-9\-]+)/i: /tag/$1/
  /v([0-9\.]+)/docs/([a-z0-9\-]+)/i: /docs/$2/
```
- Added 2 new endpoints: `POST redirects/upload` and `GET redirects/download`. These serve as an alias to current GET/POST `/redirects/json. "upload/download" naming pattern is introduced to match the convention with other resources that can be uploaded and downloaded (images, themes etc.). `/redirects/json`  endpoints will be removed in Admin API v4
- The parsing code from `custom-redirects.js` has been moved to `frontend/services/redirects/settings.js`. This location is more appropriate for this logic and eventually `custom-redirects.js` middlewear might be moved into "frontend" as this middlewear plays a role mostly effecting that area.
2020-11-04 12:08:32 +13:00
Daniel Lockyer
300145798e Updated Ghost-Admin to v3.37.1 2020-11-03 14:00:30 +00:00
Daniel Lockyer
4a270632db Updated Ghost-Admin to v3.37.0 2020-11-03 13:11:42 +00:00
Rishabh Garg
8ad11fe082
Enabled Portal (#12317)
no refs

[Portal](https://github.com/TryGhost/Portal) is a new drop-in script to make the bulk of Ghost membership features work on any theme out of the box, which was under a developer flag so far. This release removes the flag for Portal and makes it included as default for any members-enabled Ghost site. The Portal script is backward compatible with old public members script and existing Members-enabled themes should notice no change.

- Removes Portal config flag as Portal is now enabled by default
- Removes old members script as Portal is backward compatible with it
- Changes `{{content}}` helper to show default CTA in case of restricted content access
- `accent_color` setting is no more behind the dev experiment flag and included by default
- Adds migration to switch off Portal button setting for all existing sites which don't have Portal enabled in beta
2020-11-03 14:36:21 +05:30
Naz
4ebebd12d3 Removed unneeded parameter
no issue

- There is no second parameter in Promise constructor. Didn't serve any purpose as per comment - bbbd011074 (r41413465)
2020-11-03 13:38:08 +13:00
Naz
7326241718 🐛 Fixed server error for repeated order query parameter
closes #12263

- Express parses repeated query parameters as an array (req.query properties). Because there is no clear reason on why not to support this behavior extended order parameter parsing logic to handle arrays. This follows the rule of "liberal inputs, conservative outputs"
- Example supported query string for ordering can now look like: `?order=featured&order=published_at asc`, the priority of the order stays the same with the most significant appearing first and least significant last
2020-11-03 00:15:24 +13:00
naz
a2bfef53de
🐛 Fixed post resource filtering by posts_meta table fields (#12307)
refs #11572

- Filtering by fields coming from posts_meta table did not work for post resources. This was due to lack of support for these types of operations on NQL layer. The approach taken here is using same way filtering was done for many:many relations and generates a `WHERE IN` filtering clause. In the future we could look into adding preloading of 1:1 relations which should allow getting rid of `WHERE IN` in favor of `JOIN` and filtering directly by field names. 
- Changed structure of `EXPANSIONS` filter configuration. Current approach was based on "bag of all the things". Such structure will become problematic as more fields are added. For example, adding all the fields from 1:1 relation posts:posts_meta might collide with any other relations that would have similar naming like meta_description from tags table (if it were was added).
- Bumped nql version to 0.5.0. This adds filtering support to 1:1 relations
- Added filter expansions which can be unique per model Previous approach with single global expansions lookup wasn't working in case different models would need to declare expansion for same field names. Having a `filterExpansion` method per model works in a similar convention other filter related model methods do (e.g. enforcedFilters, defaultFilters)
2020-11-02 12:53:57 +13:00
Kevin Ansfield
7a74b78940 Included email-id as user variable when sending bulk emails via mailgun
no issue

- adding user variables via the mailgun API when sending emails means that events related to email have those variables attached to them
- adding the `email.id` value to user variables means we can easily associate mailgun events to emails, otherwise we'd have look up the batch via `email_batches.provider_id` then use a join to get back to the associated email
2020-10-29 16:37:42 +00:00
Daniel Lockyer
85da9bdeb2 Updated Ghost versioning lib to include all prerelease IDs
- currently if Ghost has a version of something like
  `3.37.0-pre.abc.def`, Ghost will return `3.37.0-pre.abc` as the full
  version
- this hides parts of the version which are useful for debugging
- this commit updates the logic to join together all prerelease elements
  so we keep the full string
2020-10-29 12:44:55 +00:00
Rish
c4c6bdf353 Updated mail template for member email change
no issue

The email change verification template was using the same as for `subscribe`, which did not have the right messaging that needs to be communicated about the action thats happening in this case. Updates the email template to same as what we use for email verification for support/newsletter address
2020-10-29 17:40:19 +05:30
Rish
746ccbd7ed Added custom action param for magic links
closes https://github.com/TryGhost/Ghost/issues/12253

- Allows using custom action param for requests from Portal by using a new `requestSrc` option that is passed down when a request for magic link is made via Portal
2020-10-29 13:42:52 +05:30
Rishabh Garg
0ce5f87616
🐛 Added past_due and unpaid subscriptions for members (#12301)
refs https://github.com/TryGhost/Ghost/issues/12256 , https://github.com/TryGhost/Ghost/issues/12255

Currently when listing subscriptions for Members in both the Admin and the Theme, we only show the subscriptions which have a status of trialing or active.

Based on discussion, the `unpaid` and `past_due` states on Stripe also represent owner's intention of considering a subscription as active instead of `cancelled`, so we allow any subscriptions under these 2 states to be also listed for a member and consider them as `paid`.

- Subscriptions will go into a past_due state if the payment is missed, this should be considered a grace period where the member still has access.

- After this the subscriptions will either go to the unpaid or the cancelled state - this can be configured on an account by account basis in the Stripe dashboard. `unpaid` is considered as an intention to keep the subscription to allow for re-activation later.
2020-10-27 15:19:48 +05:30
Peter Zimon
43b7c7b344
Updated members admin redirect to new page(#12296)
no refs.

- Members settings were moved to a new page [here](https://github.com/TryGhost/Ghost-Admin/pull/1736)
- updated members related redirect for from/newsletter email address change to point to new page
2020-10-22 16:10:25 +05:30
Daniel Lockyer
c68c749026 Updated Ghost-Admin to v3.36.0 2020-10-20 09:38:09 +01:00
Rishabh Garg
7182efdb88
🐛 Fixed special chars in single use token (#12290)
no refs

- The token generation logic for single use token was replacing only the first instance of + or / to make the token URL safe, instead of replacing all instances which caused a bug where token was not validated properly in case it included multiple + or / in it.

- The fix ensures replacing all the + or / in the token with URL safe _ or - so it can be properly validated via magic link
2020-10-20 11:49:20 +05:30
naz
8ddf83f3c5
Fixed "no-shadow" linting error in server modules (#12287)
refs 143921948d

- Continuation of changes started in referenced commit
2020-10-20 12:02:56 +13:00
naz
38b7f8e311
Fixed "no-shadow" linting error in server/data modules (#12288)
refs 143921948d

- Continuation of changes started in referenced commit
2020-10-20 11:56:46 +13:00
Kevin Ansfield
d08b586781 🐛 Fixed width/height attributes in gallery cards not matching resized images
closes https://github.com/TryGhost/Ghost/issues/11548

- bumps @tryghost/kg-default-cards
  - introduces a breaking change where `imageOptimization` is passed as an options argument instead of separating out individual config options
  - fixes width/height attributes for images in gallery cards being larger than the max default width used when resizing images
2020-10-19 17:05:57 +01:00
Kevin Ansfield
3148d6a946
Moved definition of default max image width to config (#12289)
no issue

- centralises definition of max width and allows customisation if needed
- allows for passing of the config value through to rendering libraries
2020-10-19 08:56:18 +01:00
Kevin Ansfield
16bfb3fa41 Added basic CRUD admin API for snippets
no issue

- standard browse/read/add/edit/destroy API endpoints for snippets resource
- updates `@tryghost/admin-api-schema` dependency to version that includes snippet definition and schemas
2020-10-16 18:02:58 +01:00
Kevin Ansfield
e5a92be932 Fixed typo in snippets permission fixtures
refs https://github.com/TryGhost/Ghost/pull/12283

- `delete` -> `destroy` for action type
2020-10-16 17:46:31 +01:00
Kevin Ansfield
15afed4b81
Added snippets permissions migration and fixtures (#12283)
no issue

- all staff users can browse/read snippets so they're usable in the editor for everyone
- only administrators, editors, and admin integrations are able to create/edit/delete snippets
2020-10-16 12:53:09 +01:00
Kevin Ansfield
74269070dd
Added migration for snippets table creation (#12282)
no issue

- minimal table structure required for the first iteration of content snippets
- snippets are stored pieces of re-usable content that could effectively be entire posts so the `mobiledoc` field length matches the `posts.mobiledoc` field length
2020-10-16 12:52:06 +01:00
Rish
a5a7515fe8 Fixed typo in private setting import
no refs

Last commit introduced a small typo in how private setting is ignored during import
2020-10-15 15:55:24 +05:30
Rish
3e0948d04e Ignored member email settings in import
no issue

- Member email settings - newsletter and support - was previously being set to null for all cases, which can lead to `null` values being set for those addresses on a successful import whereas the expected behavior is to retain the old values.
- Adds the 2 email settings to ignored settings list so they are not changed during import process
2020-10-15 15:32:05 +05:30
Rish
54e638310c Added default support address for member site data
no issue

- In a case where support address is null, member's site data won't load as there was no fallback.
- The fix adds default `noreply` as the support address for site data
2020-10-15 14:44:16 +05:30
Fabien O'Carroll
a9b3d83e00 Loaded Action model after all other models
no-issue

The Action model loops through all registered models when the file is
loaded, by loading the model last, we ensure that it can read all
models, rather than an arbitrary selection which come before it.
2020-10-14 15:11:45 +01:00
Fabien O'Carroll
0fd0527da0 Fixed creation of EmailBatch and EmailRecipient collections
no-issue
2020-10-14 15:11:45 +01:00
naz
392140cb36
Allowed for comped field when creating a member through Members API (#12278)
closes #12273

- `comped` field has been allowed when editing a member or importing from a CSV. There has been a usecase (Zapier Integration) for API client to create a member with "Complimentary" plan, which made this change necessary
- Previously the logic for comped field was to skip and continue member record creation if Stripe was not connected. Now we throw an error - same as the one we have been throwing before when stripe_customer_id field was passed in. The implication of this change is that we won't be creating any record now if comped === true and Stripe is disabled. 
- Bumped admin-api-schema-package. Contains `comped` schema change so this field gets passed through to controller
2020-10-14 13:24:09 +13:00
Daniel Lockyer
39ff4fe688 Updated Ghost-Admin to v3.35.5 2020-10-08 11:00:35 +01:00
Kevin Ansfield
8ad384d7f8 🐛 Fixed scheduled post emails pointing at /404/ for the "view online" link
no issue

When scheduling a post to publish+send the "view online" link was pointing at https://site.com/404/ rather than the published post's url.

The problem occurred because the `/schedules/` endpoint wraps it's post read+edit calls in a transaction. Context:
- when a post is published with with the "send email" option the email record is immediately generated and added to the API response, as part of the email record generation we render the email content including fetching the url for the "view online" link
- urls for all resources are handled by our `url` service, that service updates it's internal cache based upon model events such as the "edited" event triggered when a post is published
- if the posts API controller is given a transaction, the email record is also generated inside of that transaction however at this point the `url` service will not have been updated because the post record hasn't been committed meaning it has no available url for the post

Fix:
- removed the `models.Base.transaction()` wrapper around the post read+update in the `/schedules/` API controllers
- we don't need a transaction here. It was added as protection against another write request coming in between the `/schedules/` controller reading a post and publishing a post but we already have protection against that in the form of collision detection - if a write request comes in and commits between the schedules controller reading the post and updating it, the scheduler's update call will fail with a collision error at which point the scheduler itself should retry the request which could then publish the post successfully if everything else is in order
2020-10-08 10:26:49 +01:00
Daniel Lockyer
90a00ab3e5 Updated Ghost-Admin to v3.35.4 2020-10-06 09:55:49 +01:00
Kevin Ansfield
9cbeb74db0
🐛 Fixed broken embeds cards when pasting links to Wordpress sites (#12262)
closes https://github.com/TryGhost/Ghost/issues/12260

- if a card type was not explicitly chosen (i.e. a url was pasted into the editor) then abort fetching the oembed endpoint if we detect it's a `wp-json` oembed and return a bookmark card payload instead
- cleaned up an unused argument in the internal `fetchBookmarkData()` method
2020-10-06 08:44:03 +01:00
Kevin Ansfield
cde364bf27 🐛 Fixed email card replacements showing raw replacement text in emails
closes https://github.com/TryGhost/Ghost/issues/12257

- there was a destructuring problem introduced in the recent email refactor which meant the correct replacement data was not being passed over to the Mailgun provider when sending email
2020-10-05 17:24:48 +01:00
Kevin Ansfield
4f211d025d Fixed members with multiple subscriptions receiving multiple newsletters
closes https://github.com/TryGhost/Ghost/issues/12259

- adds a `DISTINCT` to the query used to fetch member rows when generating an email recipient list
- this increases query time 2.7s vs 1.6s locally with ~94k paid members but once the `members.paid` column is implemented this slow query can be removed
2020-10-05 16:53:35 +01:00
Naz
39e403d176 🐛 Fixed meta attributes calculation on post preview
closes https://github.com/TryGhost/Ghost/issues/12247

- Internal preview controller was lacking "mapping" call to post object which handled not only missing meta attribute information but lots of other mappings (e.g. users, tags, etc.)
- Have added a regression test to catch issues like this in the future
2020-10-05 17:52:40 +13:00
Sam Lord
46c20759f3 Updated Ghost-Admin to v3.35.3 2020-10-02 14:38:32 +01:00
Kevin Ansfield
8f3ab3c535 🐛 Fixed email showing as success when an email batch fails to send
no issue

- fixed passing of errors up through send/processBatch/processEmail
- fixed errant overwrite of email status with a "submitted" status after a failure had occurred
2020-10-02 14:26:57 +01:00
Kevin Ansfield
b5ffb38726 Fixed partial email batch/recipient records when email creation fails
no issue

- wrap email batch/recipient record creation in a transaction so if an error occurs during creation we're not left with a partially created batch/recipient set in the database
2020-10-02 13:47:14 +01:00
Kevin Ansfield
0f30b9f6a0 🐛 Fixed email not showing as failed if error occurs when preparing email
no issue

- if an error occurred whilst creating email batch/recipient records the email status was never updated and was left in the 'pending' status
- adjusted the error handling to update the email status and record the error message if such a scenario occurs
2020-10-02 12:40:49 +01:00
Kevin Ansfield
7b789e1cd5 🐛 Fixed newsletters being sent to Stripe customer emails in place of member emails
no issue

- the paid-member SQL query that is obtained using `models.Member.getFilteredCollectionQuery({paid: true})` can return multiple columns with the same name (eg, `email`, `name`), when that happens the last column with duplicate names "wins" and it's value is used in the resulting knex row instance
- in the `mega` service when fetching email recipient rows we ran into this problem, to avoid it we adjust the query to explicitly select only the data from the `members` table
2020-10-02 12:17:17 +01:00
Kevin Ansfield
82126f29e6 Added guard for member rows having missing data when creating email recipients
no issue

- we've had an issue with emails failing due to unexpectedly missing data when inserting email recipient rows
- added a validation check before adding recipient details along with a log so that invalid data can be investigated
2020-10-02 11:29:10 +01:00
Rish
58fda5bad2 Allowed accent color setting for site with portal flag
no issue

- We had previously allowed accent_color setting for member site settings behind portal flag, but Ghost Admin also needs the public site setting with accent color to correctly reflect the accent color when flag is switched on
- Removes deletion of accent color setting when behind the Portal flag OR dev experiment flag
2020-10-02 15:30:54 +05:30
Sam Lord
d812e57813 Updated Ghost-Admin to v3.35.2 2020-10-01 19:57:14 +01:00
Kevin Ansfield
defb43fe7a 🐛 Fixed newsletters emails having no subject
no issue

- subject was not being picked out of the message data when passing over to the mailgun-js send method
2020-10-01 19:51:01 +01:00
Sam Lord
3e0f8c1eb9 Updated Ghost-Admin to v3.35.1 2020-10-01 19:21:26 +01:00
Kevin Ansfield
e34acc31c5 🐛 Fixed newsletter email sending
no issue

- there was an typo in the recent email sending refactor that resulted in `Error: 'to' parameter is missing` errors when sending email previews and bulk emails
2020-10-01 19:03:57 +01:00
Fabien 'egg' O'Carroll
ce14d8113a
🐛 Fixed members form missing data-members-name (#12250)
refs #12249

This was incorrectly assuming the presence of the data-members-name
element in the document. By guarding against it and defaulting to
undefined, we fallback to the existing behaviour when the element is not
present.
2020-10-01 18:05:23 +01:00
Fabien 'egg' O'Carroll
244704156c
Updated all Origin header checks to handle 'null' (#12246)
closes #12244

As per RFC 6454 the Origin header MUST be set to the string 'null' when
in a "privacy-sensitive" context. We were not handling this string and
this was causing errors. This commit updates all checks of the 'Origin'
header to treat the value 'null' as if the header was not present.

ref: https://tools.ietf.org/html/rfc6454#section-7.3
2020-10-01 09:37:22 +01:00
Rishabh Garg
eb4933d27e
Added extra settings to member auth emails (#12238)
no issue

- Adds accent color, site domain and url to member auth emails for customization
2020-10-01 12:31:36 +05:30
Sam Lord
4ad4892d3d Updated Ghost-Admin to v3.35.0 2020-09-30 11:00:14 +01:00
Fabien O'Carroll
fd5194ea46 Updated magic link expiry to 24 hours
no-issue

As discussed with @JohnONolan 24 hours is the preferred expiry for magic links
2020-09-30 10:53:35 +01:00
Fabien 'egg' O'Carroll
c19d282a51
Ensured Admin API members resource only returns known fields (#12240)
refs #12055

As part of the work in TryGhost/Members#206 we load the stripeCustomers relation on the member model, and we do not want this to be part of the API response. The changes here include a refactor but the main thing is that the serialized object is explicit and does not include unexpected or unknown fields.

* Moved mapMember out of mapper file

This cleans up the serializer a bit by keeping it's functionality all in
one place, rather than a shared mapper file

* Refactored members controller to return models

Previously the controller was calling toJSON, which is serialization,
this updates the controller to only deal with models, leaving all of the
serialization to the serializer!

* Refactored members serializer

This adds typings to all of the methods/functions in the serializer, as
well as making the serializating explicit, rather than returning the
result of toJSON, we explicitly set the properties we expect to be on
the output object. This protects us against accidental API changes in
the future.
2020-09-30 10:22:22 +01:00
naz
071bf8edb5
Extracted Admin API JSON schemas into @tryghost/admin-api-schema (#12237)
closes https://github.com/TryGhost/Ghost/issues/10628

- JSON Schemas were extracted into a separate module to allow other clients to reuse them (for example documentation). Having them in a separate package also slims down the amount of code needed to be maintained in the core.
- Updated canary API input validators to use admin-api-schema module
- Removed canary schemas that moved into admin-api-schema package
- Updated v2 API input validators to use admin-api-schema package
- Removed v2 schemas that moved into admin-api-schema package
- Updated tests to contain needed information in apiConfig to pick up correct validation
- Added @tryghost/admin-api-schema package dependency
2020-09-30 16:25:12 +13:00
Kevin Ansfield
932def7e6d Fixed replacement strings showing when viewing sent emails
refs https://github.com/TryGhost/Ghost/pull/12232

When viewing sent emails in Ghost's admin area, it displays the `html` field directly from the `email` relation loaded with the post. Since the `mega` refactor we now store raw content in that field rather than sanitized "preview" content so it's necessary to modify the API output to match the old behaviour.

- use the API output serializers to parse replacements in email content and replace with the desired fallback or empty string
2020-09-29 17:39:22 +01:00
Kevin Ansfield
474e6c4c45 Refactor mega service to use stored email content and batch/recipient records
no issue

- store raw content in email record
  - keep any replacement strings in the html/plaintext content so that it can be used when sending email rather than needing to re-serialize the post content which may have changed
- split post email serializer into separate serialization and replacement parsing functions
  - serialization now returns any email content that is derived from the post content (subject/html/plaintext) rather than post content plus replacements
  - `parseReplacements` has been split out so that it can be run against email content rather than a post, this allows mega and the email preview service to work with the stored email content
- move mailgun-specific functionality into the mailgun provider
  - previously mailgun-specific behaviour was spread across the post email serializer, mega, and bulk-email service
  - the per-batch `send` functionality was moved from the `bulk-email` service to the mailgun provider and updated to take email content, recipient info, and replacement info so that all mailgun-specific headers and replacement formatting can be handled in one place
  - exposes the `BATCH_SIZE` constant because batch sizes are limited to what the provider allows
- `bulk-email` service split into three methods
  - `send` responsible for taking email content and recipients, parsing replacement info from the email content and using that to collate a recipient data object, and finally coordinating the send via the mailgun provider. Usable directly for use-cases such as test emails
  - `processEmail` takes an email ID, loads it and coordinates sending related batches concurrently
  - `processEmailBatch` takes an email_batch ID, loads it along with associated email_recipient records and passes the data through to the `send` function, updating the batch status as it's processed
  - `processEmail` and `processEmailBatch` take IDs rather than objects ready for future use by job-queues, it's best to keep job parameters as minimal as possible
- refactored `mega` service
  - modified `getEmailData` to collate email content (from/reply-to/subject/html/plaintext) rather than being responsible for dealing with replacements and mailgun-specific replacement formats
    - used for generating email content before storing in the email table, and when sending test emails
    - from/reply-to calculation moved out of the post-email-serializer into mega and extracted into separate functions used by `getEmailData`
  - `sendTestEmail` updated to generate `EmailRecipient`-like objects for each email address so that appropriate data can be supplied to the updated `bulk-email.send` method
  - `sendEmailJob` updated to create `email_batches` and associated `email_recipients` records then hand over processing to the `bulk-email` service
  - member row fetching extracted into a separate function and used by `createEmailBatches`
  - moved updating of email status from `mega` to the `bulk-email` service, keeps concept of Successful/FailedBatch internal to the `bulk-email` service
2020-09-29 17:17:54 +01:00
Kevin Ansfield
d34a3263e8 Store email batch and recipient records when sending newsletters (#12195)
requires https://github.com/TryGhost/Ghost/pull/12192

- added initial `EmailBatch` and `EmailRecipient` model definitions with defaults and relationships
- added missing `post` relationship function to email model
- fetch member list without bookshelf
    - bookshelf can add around 3x overhead when fetching the members list for an email
    - we don't need full members at this point, only having the data is fine
    - if we need full models later on we can push the model hydration into background jobs where recipient batches are fetched ready for an email to be sent
    - bookshelf model instantiation of many models blocks the event loop, using knex directly keeps concurrent requests fast
    - adds `getFilteredCollectionQuery` method to base model to facilitate getting a knex query based on our normal model filters along with transaction/forUpdate applied
- store recipient list before sending email
    - chunk already-fetched members list into batches and insert records into the `email_recipients` table via knex
    - chunked into batches of 1000 to match the number of emails that Mailgun accepts in a single API request but this may not be the absolute fastest batch size for recipient insertion:
        | Batch size | Batch time | Total time |
        | ---------- | ---------- | ---------- |
        |        500 |       20ms |     4142ms |
        |       1000 |       50ms |     4651ms |
        |       5000 |      170ms |     3540ms |
        |      10000 |      370ms |     3684ms |
    - create an email_batch record before inserting recipient rows so we can effeciently fetch recipients by batch and store the overall batch status
2020-09-29 17:17:54 +01:00
Rish
8e0976fc8f Allowed accent color in members site data
no issue

- The accent_color setting was being removed from members site data when behind portal flag
- The accent color setting is now allowed in members site data for all cases as it doesn't make any sense to remove it specifically from here where we already have all the other Portal settings included which is a dev/portal flag feature anyways
2020-09-29 17:23:33 +05:30
Fabien 'egg' O'Carroll
854a41e556
🐛 Fixed CSV import json-schema email validation (#12239)
no-issue

By using the "email" validation, we were validating emails in CSV
imports using a different validator to the rest of the API. AJV's built
in email validation was failing on emails with "special" characters,
such as letters with an umlaut above them.

This commit brings the validation for CSV imports in line with the rest
of the API.
2020-09-29 12:51:35 +01:00
Peter Zimon
4e45373730 Updated Webhook API error message
refs 6f1abc610a

- updated missing Webhook API error message when integration is missing
2020-09-28 13:24:47 +02:00
Naz
50f30c4f66 Fixed minor formatting in i18n file 2020-09-28 14:21:29 +13:00
Daniel Lockyer
685e2782b7 Merged 3.34.1 into master
v3.34.1

* tag '3.34.1':
  v3.34.1
  Updated Ghost-Admin to v3.34.1
  🐛 Fixed email verification mails not sent
2020-09-24 19:28:22 +01:00
Daniel Lockyer
4cb228fab0 Updated Ghost-Admin to v3.34.1 2020-09-24 19:26:52 +01:00
Rish
82399393b8 🐛 Fixed email verification mails not sent
no issue

- Email ownership verification emails for support/from address was not using the updated magic link service syntax
2020-09-24 19:10:57 +01:00
Rish
03968219a7 🐛 Fixed email verification mails not sent
no issue

- Email ownership verification emails for support/from address was not using the updated magic link service syntax
2020-09-24 23:39:25 +05:30
Kevin Ansfield
1dfaf8c5f8
Added migration to store from/reply-to values in email table (#12204)
no issue

The email table should be a reference for all data that was used when sending an email. From and Reply-to addresses can change over time and we don't have any other reference for their value at the time of sending an email so we should store them alongside the email content.

- schema updated with `from` and `reply_to` columns
- both are set to `nullable` because we don't have historic data (can be populated and changed in later migrations if needed)
- neither `from` or `reply_to` have `isEmail` validations because they can have name+email in an email-specific format
- will help keep concerns separated in the future. `mega` service can deal with all of the email contents/properties, and the `bulk-email` service's concerns are then only email sending and any provider-specific needs
2020-09-24 08:20:10 +01:00
Rish
78379e76c1 Renamed membersjs script name to portal
no issue

- The members.js package was renamed as `@tryghost/portal`, which also updated the unpkg link for the script
- Updates the unpkg script for portal to use the new package name and path
2020-09-24 12:08:51 +05:30
Naz
43153ba31e Disabled changing webhook's integration_id through PUT API
refs #12033

- Allowing to change parent integration opens up possible security holes and has no clear usecase at the moment. After a webhook record is created it should not be possible to change parent integration.
- Had do partially duplicate JSON schema definition from webhooks definition as there is no proper composition technique available in current version of JSON Schema.
2020-09-24 16:24:02 +12:00
Naz
6f1abc610a Added check for parent integration_id when creating a webhook
refs https://github.com/TryGhost/Ghost/issues/12033
refs https://github.com/TryGhost/Ghost/issues/10567

- Creating a webhook without valid parent integration leads to orphaned webhook records, which shoult not ever happen
- This scenario is only possible for non-integration authentication,
because in case of integration being authenticated it's id is
automatically assigned to creatd webhook
2020-09-24 15:09:51 +12:00
naz
bbcc83dadb
Added support for ordering Post API resources by fields coming form posts_meta table (#12226)
refs #11729

- When ordering is done by fields from a relation (like post's `meta_title` that comes form `posts_meta` table), Bookshelf does not include those relations in the original query which caused errors. To support this usecase added a mechanism to detect fields from a relation and load those relations into query. 
- Extended ordering to include table name in ordered field name.  The information about the table name is needed to avoid using `tableName` within pagination plugin and gives path to having other than original table ordering fields (e.g. order by posts_meta table fields)
- Added test case to check ordering on posts_meta fields
- Added support for "eager loading" relations. Allows to extend query builder object with joins to related tables,
which could be used in ordering (possibly in filtering later). Bookshelf does not support ordering/filtering by proprieties coming from relations, that's why this kind of plugin and query expansion is needed
- Added note about lack of support for child relations with same property names.
2020-09-24 13:32:40 +12:00
Kukhyeon Heo
635d26469f
Fixed settings reinit to only emit changed settings events (#12184)
closes #12038

Previously we were emitting changed events for _all_ settings which would
cause any listeners for those to be triggered, this ensures that listeners are
only triggered if the corresponding setting, _did_ in fact change.
2020-09-23 14:35:03 +01:00
Fabien O'Carroll
171908cefa Added support for filtering Members CSV exports
no-issue

Adding these options to the endpoint will pass them to the model layer,
allowing us to filter members CSV exports
2020-09-23 14:29:15 +01:00
Léo Bourrel
87855075c0
Added support for data-members-name in themes (#12191)
no-issue

This change allows themes to collect member names on signup with use of the
`data-members-name` attribute on an input element in the `data-members-form`
2020-09-23 13:59:26 +01:00
Naz
bff2fd5f3b Corrected comment describing redirect behavior
no issue

- The redirect is done from `/feed` to `/rss`
2020-09-23 00:37:55 +12:00
Daniel Lockyer
40cb03d9d1 Updated Ghost-Admin to v3.34.0 2020-09-22 11:25:12 +01:00
Nazar Gargol
84c8bcc457 Extracted ordering into separate bookshelf plugin
refs #11729

- Having a plugin allows to reason about ordering better and gives way
to further extraction away form the core.
- Just like with filtering, ordering falls into similar category of having effect on knex'es query builder object extension through additional statements - ORDER BY in this case.
- Because there were bugs associated with use of permittedAttributes inside of `parseOrderOption` method, introduced separate `orderAttributes` function which returns only those fields which are orderable (https://github.com/TryGhost/Ghost/issues/11729#issuecomment-685740836)
2020-09-22 16:27:12 +12:00
Nazar Gargol
32b37d7ba8 Improved error messaging for password reset process
refs #11878

- When password reset link is invalid previous messaging left the user
without clear information about why the reset failed and what they could do about it.
- Updated messaging around password reset tokens including detection of
when password token has invalid structure, has expired or has already
been used
2020-09-22 15:45:19 +12:00
Kevin Ansfield
6dc8d91ace Revert "Store email batch and recipient records when sending newsletters (#12195)"
This reverts commit 80af56b530.

- reverting temporarily so that all associated functionality can be merged in a single release
- creating email batch/recipient records without using them would cause inconsistent data
2020-09-21 17:02:59 +01:00
Fabien 'egg' O'Carroll
39dd492927
Updated magic link expiry information to 4 hours (#12224)
no-issue

Since TryGhost/Ghost#12218 magic links now have
an expiry of 4 hours.
2020-09-21 16:08:21 +01:00
Kristian Freeman
aca94e65c3
Added support for Stripe promo codes in config (#12149)
closes #12111 

* Added default value in defaults.json
* Passed config to members-api module

refs: https://github.com/TryGhost/Members/pull/194
2020-09-21 13:15:41 +01:00
Nikolay A. G
e5f2fa650b
Removed unmatched closing span tag on AMP pages (#12183)
no issue

- The unmatched closing </span> tag is ignored by the browser anyway and is not semantically correct
- Checked the history if there was any possible purpose behind it in the past but doesn't seem like it
2020-09-21 12:53:27 +12:00
Fabien 'egg' O'Carroll
7c5a3bb537
Updated magic links to use shorter, single us, longer lived tokens (#12218)
no-issue

* Added SingleUseTokenProvider to members service

This implements the TokenProvider interface required by members-api to
generate magic links. It handles checking if the token is expired and
pulls out any associated data.

Future improvments may include the email in the error for expired
tokens, which would make resending a token simpler.

* Passed SingleUseTokenProvider to members-api

This sets up the members-api module to use the new single use tokens

* Installed @tryghost/members-api@0.30.0

This includes the change to allow us to pass a token provider to the members-api
2020-09-18 17:32:18 +01:00
Fabien 'egg' O'Carroll
812e4b682f
Added SingleUseToken model (#12215)
no-issue

This is a model for the tokens table, which handles the single use
aspect by customising the `findOne` method to automatically destroy the
model after reading from it
2020-09-18 15:05:56 +01:00
Fabien O'Carroll
fe1e480e2e Updated getMagicLink call to handle Promise return
no-issue

refs: https://github.com/TryGhost/Members/commit/63942f03

The above commit updated all magic-link methods to be async, this change
ensures that we will handle a Promise return when it's updated in Ghost.
`await` has no effect on non Promise return values so it's safe to add
this now.
2020-09-18 10:49:18 +01:00
Fabien O'Carroll
a862a58b5f Updated the tokens table to have 32 char tokens
no-issue

After discussion with Matt, we decided that 192 bits for the token is a
good number, as it has no padding when base64 encoded and is more secure
than 128 bits, whilst still a managable size.
2020-09-17 17:33:43 +01:00
Fabien O'Carroll
01e62e3d79 Added tokens table to the database
no-issue

This is a table to store single use tokens for use in magic links, the
columns are as simple as possible at the moment and are designed as:
id - standard ObjectID like all of our tables
token - 128bit base64 encoded string
data - arbitrary data to store against the token
created_at - timestamp to allow for expiry to be implemented for tokens
2020-09-17 16:08:35 +01:00
Fabien O'Carroll
ed7adf16a2 Added migration util for adding new tables
no-issue

This pulls out some commonly used logic into a util for future use
2020-09-17 16:08:35 +01:00
Rish
ac146f1e3b Updated ownership verification email flow to use different From address
no issue

- In a recent change to ownership verification email flow, we changed the FROM address of ownership verification mails to use the same email as the one we are verifying, aka TO address.
- Email clients like Gmail flags off such emails as possible spam
- Fix updates the `FROM` address to `noreply@domain.com` where domain.com is domain for TO address
-  In case the TO is already noreply@domain.com, we use no-reply@domain.com to bypass the same address restriction.
2020-09-17 19:52:51 +05:30
Rish
7a3839fb14 Added new portal config flag
no issue

- The new Portal config flag allows switching on Portal conditionally with config
- The dev experiment flag still works for enabling Portal
- The flag currently defaults to `false` as Portal is still a beta feature and switched off by default
- We expose it on the admin api config endpoint so that the Ghost-Admin client can use it to conditionally render Portal settings
2020-09-17 19:40:34 +05:30
Rish
49abeaf398 Added blog domain for default support address
no issue

- Members site data was not appending blog domain for default support address which is `noreply`
- The change allows Portal to use default support address correctly
2020-09-17 12:33:16 +05:30
Peter Zimon
ff2d9ebdfd Updated copy on default content CTA
no refs.
- updated button copy for logged in free members on default content CTA
2020-09-16 16:41:38 +02:00
Nazar Gargol
46932c6d75 Removed unused messages from i18n file
no issue

- These messages are never used throughout the system. Removed to keep things tidy.
2020-09-16 15:46:22 +12:00
Kevin Ansfield
80af56b530
Store email batch and recipient records when sending newsletters (#12195)
requires https://github.com/TryGhost/Ghost/pull/12192

- added initial `EmailBatch` and `EmailRecipient` model definitions with defaults and relationships
- added missing `post` relationship function to email model
- fetch member list without bookshelf
    - bookshelf can add around 3x overhead when fetching the members list for an email
    - we don't need full members at this point, only having the data is fine
    - if we need full models later on we can push the model hydration into background jobs where recipient batches are fetched ready for an email to be sent
    - bookshelf model instantiation of many models blocks the event loop, using knex directly keeps concurrent requests fast
- store recipient list before sending email
    - chunk already-fetched members list into batches and insert records into the `email_recipients` table via knex
    - chunked into batches of 1000 to match the number of emails that Mailgun accepts in a single API request but this may not be the absolute fastest batch size for recipient insertion:
        | Batch size | Batch time | Total time |
        | ---------- | ---------- | ---------- |
        |        500 |       20ms |     4142ms |
        |       1000 |       50ms |     4651ms |
        |       5000 |      170ms |     3540ms |
        |      10000 |      370ms |     3684ms |
    - create an email_batch record before inserting recipient rows so we can effeciently fetch recipients by batch and store the overall batch status
2020-09-14 15:40:00 +01:00
Daniel Lockyer
1221306212 Updated Ghost-Admin to v3.33.0 2020-09-14 12:39:47 +01:00
Kevin Ansfield
76c1b60a4d
Added schema+migration for email_{batches,recipients} tables (#12192)
no issue

We want to store a list of recipients for each bulk email so that we have a consistent set of data that background processing/sending jobs can work from without worrying about moving large data sets around or member data changing mid-send.

- `email_batches` table acts as a join table with status for email<->email_recipient
  - stores a provider-specific ID that we get back when submitting a batch for sending to the bulk email provider
  - `status` allows for batch-specific status updates and picking up where we left off when submitting batches if needed
  - explicitly tying a list of email recipients to a batch allows for partial retries
- `email_recipients` table acts as a join table for email<->member
  - `member_id` does not have a foreign key constraint because members can be deleted but does have an index so that we can efficiently query which emails a member has received
  - stores static copies of the member info present at the time of sending an email for consistency in background jobs and auditing/historical data
2020-09-14 12:21:58 +01:00
naz
cbdc91ce48
Added Location header to API's POST request responses (#12186)
refs #2635

- Adds 'Location' header to endpoints which create new resources and have corresponding `GET` endpoint as speced in JSON API - https://jsonapi.org/format/#crud-creating-responses-201. Specifically:
    /posts/
    /pages/
    /integrations/
    /tags/
    /members/
    /labels/
    /notifications/
    /invites/

- Adding the header should allow for better resource discoverability and improved logging readability
- Added `url` property to the frame constructor. Data in `url` should give enough information  to later build up the `Location` header URL for created resource.
- Added Location header to headers handler. The Location value is built up from a combination of request URL and the id that is present in the response for the resource. The header is automatically added to requests coming to `add` controller methods which return `id` property in the frame result
- Excluded Webhooks API  as there is no "GET" endpoint available to fetch the resource
2020-09-14 22:33:37 +12:00
Rish
7b5401b268 Fixed loading default template style for cta behind flag
refs e3a0bb535f

- The default style template was incorrectly loaded from the stripe config check instead of dev config flag
2020-09-11 11:10:24 +05:30
Daniel Lockyer
453faff41e Merged 3.32.2 into master
v3.32.2

* tag '3.32.2':
  v3.32.2
  Updated Ghost-Admin to v3.32.2
2020-09-10 17:02:50 +01:00
Daniel Lockyer
6e86bbf8e0 Updated Ghost-Admin to v3.32.2 2020-09-10 16:55:17 +01:00
Talha
afa976066a
🐛 Fixed table constraint error when updating member's email with an already existing email (#12178)
closes #12045

- When member's email is updated to an already existing email of different member it caused table's unique constraint error, which was not handled properly. 
- Added handling for this error similar to one in members `add` method.
2020-09-10 16:03:57 +12:00
Nazar Gargol
5582d030e3 Added routes.yaml content checksum storage to the db
closes #11999

- When the routes.yaml file changes (manually or through API) we need
to store a checksum to be able to optimize routes reloads in the future
- Added mechanism to detect differences between stored and current routes.yaml hash value
- Added routes.yaml sync on server boot
- Added routes.yaml handling in controllers
- Added routes hash synchronization method in core settings. It lives in core settings
as it needs access to model layer. To avoid coupling with the frontend settings it accepts
a function which has to resolve to a routes hash
- Added note about settings validation side-effect. It mutates input!
- Added async check for currently loaded routes hash
- Extended frontend settings loader with async loader. The default behavior of the loader is
to load settings syncronously for reasons spelled in 0ac19dcf84
To avoid blocking the eventloop added async loading method
- Refactored frontend setting loader for reusability of  settings file path
- Added integrity check test for routes.yaml file
2020-09-10 10:54:57 +12:00
Talha
c8a6939b2e
🐛 Fixed error caused by accepting invitation with existing email (#12172)
closes #12060

- A 500 error what happening when invited user provided an email that is associated with an existing user
- Additional validation for existing email address was added to prevent invalid data hitting db constraint error
2020-09-09 11:58:53 +12:00
Rish
3c7a95aefa Added members support address to site data
refs https://github.com/TryGhost/members.js/issues/90

- Includes new members_support_address in member site data endpoint for Portal
2020-09-08 20:48:57 +05:30
Rishabh Garg
e3a0bb535f
Added default CTA to content helper (#12157)
no issue

ATM users have to add logic to their themes in order to automatically hide restricted content. The {{content}} helper is updated to return a default CTA box instead of the post content for restricted posts with default static text using site's accent color and opening Portal for relevant action. This is currently behind the dev experiment flag.

- Adds new default content helper template in case of restricted content
- Updates content helper to trigger new CTA template in case of restricted content
2020-09-08 12:49:36 +05:30
Daniel Lockyer
8eda575fac Updated Ghost-Admin to v3.32.1 2020-09-07 12:06:05 +01:00
Daniel Lockyer
a97c1e232f Updated Ghost-Admin to v3.32.0 2020-09-07 11:54:59 +01:00
naz
fcc6401445
Added routes_hash setting (#12171)
refs #11999

- The `routes_hash` setting will be used during the boot process to update the hash
of currently loaded routes.yaml file in case it's different from last restart
2020-09-07 21:54:55 +12:00
Nazar Gargol
4606c93e4f Refactored headers function to use async/await
no issue

- The async/await syntax makes it easier to reason about the code. Because adding 'Location' header is in the works it's a prep-work in a sense
2020-09-07 15:36:06 +12:00
Rish
2d2fa1a0ba Fixed missing subscription data in member update endpoint
closes https://github.com/TryGhost/members.js/issues/94

- The member-api package was recently updated to work directly with models and needs explicit `withRelated` options to attack relations
- Without options, the endpoint was returning the default member data without subscriptions attached, which in Portal showed paid member as free
- Fix updates the middleware for updating member data to correctly pass the relations needed to populate the member
2020-09-04 17:11:04 +05:30
Rish
dd6ac57aca Fixed missing domain for default support address
no issue

- By default for new sites, support address is set same as from address to `noreply` , with full email address using the domain for `@`
- For newsletter emails, the support address was missing the default site domain to be added to address if its `noreply`
- Fix updates the support address to use the same format as from address and add relevant domain for default case
2020-09-03 16:34:47 +05:30
Daniel Lockyer
91bca9f109 Merged 3.31.5 into master
v3.31.5

* tag '3.31.5':
  v3.31.5
  Updated Ghost-Admin to v3.31.5
  🐛 Fixed incorrect status used for trial subscription query
2020-09-02 13:18:36 +01:00
Daniel Lockyer
25a994ef55 Updated Ghost-Admin to v3.31.5 2020-09-02 13:13:47 +01:00
Kevin Ansfield
52495fa929 🐛 Fixed incorrect status used for trial subscription query
refs 96c18edc82

- replaced `trialed` with `trialing` to correctly match the status used in Stripe
2020-09-02 13:11:27 +01:00
Kevin Ansfield
936d7a315a 🐛 Fixed incorrect status used for trial subscription query
refs 96c18edc82

- replaced `trialed` with `trialing` to correctly match the status used in Stripe
2020-09-02 13:07:48 +01:00
Daniel Lockyer
7309cc1ddb Merged 3.31.4 into master
* tag '3.31.4':
  v3.31.4
  Updated Ghost-Admin to v3.31.4
  🐛 Fixed missing member email on unsubscribe page
  🐛 Fixed members with Stripe trials not receiving paid-members newsletters
2020-09-02 12:31:15 +01:00
Daniel Lockyer
1b01f166f0 Updated Ghost-Admin to v3.31.4 2020-09-02 12:27:08 +01:00
Rish
7a2b7582f1 🐛 Fixed missing member email on unsubscribe page
no refs

- The `update` method in members-api package was edited to return Model object instead of JSON directly - TryGhost/Members@a28bcc5
- This unsubscribe handler was returning the raw member object returned from `update` method, which is now a model object and not able to access `member.email`
- Fix updates the unsubscribe request handler to return the member JSON again
2020-09-02 12:24:33 +01:00
Kevin Ansfield
5aadf1cf00 🐛 Fixed members with Stripe trials not receiving paid-members newsletters
no issue

- members who have trial subscriptions added directly via Stripe will have a status of `"trialed"` in their Ghost subscription
- the `paid: true` filter was not taking that into account meaning trial users were not receiving newsletters sent to paid members even though they have a "paid" subscription
2020-09-02 12:24:27 +01:00
Rish
696e60dd51 🐛 Fixed missing member email on unsubscribe page
no refs

- The `update` method in members-api package was edited to return Model object instead of JSON directly - TryGhost/Members@a28bcc5
- This unsubscribe handler was returning the raw member object returned from `update` method, which is now a model object and not able to access `member.email`
- Fix updates the unsubscribe request handler to return the member JSON again
2020-09-02 12:03:10 +05:30
Kevin Ansfield
96c18edc82 🐛 Fixed members with Stripe trials not receiving paid-members newsletters
no issue

- members who have trial subscriptions added directly via Stripe will have a status of `"trialed"` in their Ghost subscription
- the `paid: true` filter was not taking that into account meaning trial users were not receiving newsletters sent to paid members even though they have a "paid" subscription
2020-09-02 00:34:26 +01:00
Rish
806d0f8e21 Updated sender address for ownership verification emails to new address
no issue

- We used existing "from" address as sender for mails sent to new email address for verification, but that breaks the flow to update if the current "from" address has DMARC policy set.
- This updates the flow to always send the ownership verification email TO the new address and FROM the new address which both verifies the email deliverability for new address and ownership
2020-08-31 18:09:38 +05:30
Rish
8d022ecfb5 Updated newsletter emails to include reply-to address
no issue

- The newsletter emails are sent out with `from` address as sender
- The new `members_reply_address` setting is now used to set reply-to address for emails, which can be either newsletter or support address
2020-08-31 18:09:38 +05:30
Rish
ca8b77182c Updated sender address for member auth emails to support address
no issue

- Member auth emails were previously using the `from` address as sender
- New `members_support_address` was introduced with default as original "from" address
- Auth emails use the new support address as sender
2020-08-31 18:09:38 +05:30
Rish
cbb3664bdd Added handling for new members support/reply email addresses
no issue

- Updated magic link generation and validation methods for email update API to handle new support address
- Updated importer to ignore the new support address as it can only be updated via verification
- Updated members service to listen on settings edit for new support/reply address fields as well
- Updated tests to include the new settings
2020-08-31 18:09:38 +05:30
Rishabh Garg
bc2f6eb8e9 Added migrations for support and reply email address setting (#12163)
no issue

- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account, public setting
- `members_reply_address` - Where you receive responses to newsletters
2020-08-31 17:16:53 +05:30
Daniel Lockyer
4e3bc19b24 Updated Ghost-Admin to v3.31.3 2020-08-31 10:37:01 +01:00
Rish
8cf9dc05e3 Revert "Added migrations for support and reply email address setting (#12163)"
This reverts commit 2216019a86.
2020-08-31 14:25:29 +05:30
Nazar Gargol
eb8c0fcff9 🐛 Fixed Tags API v2 to return posts count
closes #12167

- Tags API v2 was ignoring `count.posts` include parameter.
- Regression was introduced with a3f693b472
- Introduced regression tests across all Content API versions to avoid similar bug in the future
2020-08-31 18:46:35 +12:00
Rishabh Garg
2216019a86
Added migrations for support and reply email address setting (#12163)
no issue

- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account
- `members_reply_address` - Where you receive responses to newsletters
2020-08-27 20:27:50 +05:30
Nazar Gargol
e8248cdc9e Refactored members imporeter in preparation for jobs
no issue

- This refactor extracts labels related code into a separate module for easier reuse by the "job-aware" batched importer
2020-08-27 21:47:00 +12:00
Nazar Gargol
262d33b73c Improved import_label creation logic
no issue

- When an import was done and there were no "global labels" present Ghost created generic `import-[data]` label which later helped to find a specific batch of imported data
- It did not make sense to create such generic label when user provided their own unique label
- The rules that work now are:
1. When there is no global provided Ghost generates on and removes it in case there are no imported records
2. When there is a unique new global label provided no new label is generated, but the label stays even if there are no imported records
2020-08-27 19:07:27 +12:00
Kevin Ansfield
f30d3cd2c2 Switched to bookshelf count in getTotalMembers()
no issue

- tested performance between knex raw, knex `count()` and bookshelf `count()` and found no difference over 1000 iterations of each (each ~19,500ms +- 500ms for 104k members locally)
- switched to using bookshelf as the code is the simplest
2020-08-27 01:52:36 +01:00
Kevin Ansfield
a995e9cc89 Allowed custom and search queries to be used with .findAll()
no issue

- extract filtering of an collection into a separate function
- use extracted function in `findAll()` so that it's query behaviour matches `findPage()`
2020-08-27 01:09:07 +01:00
Kevin Ansfield
5e64f113d5 Skipped separate count query in .findPage() for limit="all" requests
no issue

- for large result sets or complex queries the count query itself can be quite time consuming
- when `limit: 'all'` is passed as an option there's no need to perform a separate count query because we can determine the pagination data from the final result set
- skipped count query when `limit: 'all'` option is present
- re-ordered comments to be closer to the code they reference (ie, why we have our own count query instead of Bookshelf's `.count()`
2020-08-27 01:09:07 +01:00
Nazar Gargol
9fe5007929 Removed date handling validation in members importer logic
no issue

- This is handled on input sanitization layer with date
format check in JSON schema validation, so there's no need to do this
check again in the importer.
2020-08-26 17:25:24 +12:00
Nazar Gargol
544382b651 Updated boolean values validations for members uploads
no issue

- When the value is part of `enum` the `null` value is no longer
recognized by `null` type in JSON schema validator.
2020-08-26 17:23:36 +12:00
Nazar Gargol
95044e3ba0 Added error handling for failed member imports
no issue

- When bulk insert fails there is no transactional logic to revert
related records form being inserted. Also, previously there were no
attempts to "retry" the insert.
- To avoid complex retry logic, an iterative one-by-one insert retry
approach was taken. If this becomes a bottleneck in the future, the
retry algorithm could be improved.
- To avoid a lot of code duplication refactored model's `bulkAdd` & `bulkDestroy`
methods to use 'bulk-operations' module.
- Updated error handling and logging for bulk delete operations. It's very
unlikely for error to happen here,  but still need to make sure there is
a proper logging in place to trace back the failure.
- Added debug logs. This should improve debugging experience and
performance measurements.
- Added handling for unrecognized errors. Handling inspired by current unrecognized
error handling by ghost importer -10e5d5f3d4/core/server/data/importer/importers/data/base.js (L148-L154)
2020-08-26 17:11:35 +12:00
Daniel Lockyer
d9700ba9bc Updated Ghost-Admin to v3.31.2 2020-08-25 12:57:57 +01:00
Nazar Gargol
45bca2b104 Fixed linting for await statements
refs 9d65de4b10
2020-08-25 19:30:02 +12:00
Nazar Gargol
9d65de4b10 Removed created import labels when no records were imported
no issue

- When no members are succesfully imported through CSV import process the import label should not be created. Otherwise after multiple failed attempts to import there are orphaned labels in the system
2020-08-25 19:23:05 +12:00
Kevin Ansfield
e07e0a17c6 🐛 Fixed bookmark card display in member emails when using Gmail+Chrome
no issue

- updates `@tryghost/kg-default-cards` which contains two fixes
  - removes email-specific output being added to post html (had no visual impact due to use of conditional comments but keeps rendered html smaller+cleaner)
  - adds a background-url style to the thumbnail container to give two options for styling
- updates member email template styling to hide the `<img>` element in bookmark cards and use a background image instead to get consistent rendering across email clients
2020-08-24 18:21:09 +01:00
Daniel Lockyer
24c5c32fa8 Updated Ghost-Admin to v3.31.1 2020-08-24 16:29:08 +01:00
Daniel Lockyer
50a10a4903 Updated Ghost-Admin to v3.31.0 2020-08-24 11:33:22 +01:00
Peter Zimon
75f5ab637d Added minor member newsletter customisations via config
no issue
- adds a `members:emailTemplate` config object
  - `showSiteHeader` - defaults to `true`, shows the site title and icon in member emails
  - `showPoweredBy` - defaults to `false`, adds a "Publish with Ghost" button to member email footer
- updates member newsletter email template with hideable site header and "powered by" badge
2020-08-24 10:44:22 +01:00
Rish
297b63edce 🐛 Fixed response for members API update endpoint
closes https://github.com/TryGhost/members.js/issues/87

- The `update` method in members-api package was edited to return Model object instead of JSON directly [here](a28bcc5b2a)
- This caused the update member API on member endpoint to return partial response only as most properties couldn't be fetched
- Fix updates the middleware to correctly call `toJSON` before formatting response
2020-08-24 13:38:21 +05:30
Nazar Gargol
91d1417df2 Fixed email newsletter error when post has no content
closes #12158
2020-08-24 19:46:46 +12:00
Daniel Lockyer
e56921eeb7 Fixed merging of 3.30.2 into master
v3.30.2

* tag '3.30.2':
  v3.30.2
  Updated Ghost-Admin to v3.30.2
2020-08-21 12:20:51 +01:00
Daniel Lockyer
0ab190e83c Merged 3.30.2 into master
* 3.30.2:
  Bumped @tryghost/members-api to 0.28.1 in lockfile
  Bumbed @tryghost/members-api to 0.28.1
  🐛 Fixed unable to delete member when stripe is connected
2020-08-21 12:06:01 +01:00
Daniel Lockyer
e51ab7707d Updated Ghost-Admin to v3.30.2 2020-08-21 12:03:37 +01:00
Rish
87e8298ca7 🐛 Fixed unable to delete member when stripe is connected
closes https://github.com/TryGhost/Ghost/issues/12150

- The `destroy` method on members-api method was called incorrectly with not enough params
- It expects both `data` and `options` passed down [here](https://github.com/TryGhost/Members/blob/master/packages/members-api/lib/users.js#L14)
- Missing `options` in method call throws error as we read `cancelStripeSubscriptions` property on options if stripe is connected
- Fix passes both data and options to the destroy method
2020-08-21 11:13:07 +01:00
Rishabh Garg
c7f3837c54
Added admin endpoint for editing member subscription (#12145)
refs #12127

- Adds new `editSubscription` endpoint for members admin API which allows updating individual subscription for a member - `PUT /members/:id/subscriptions/:subscription_id/`
- `editSubscription` has same permissions as member's `edit` endpoint
- Currently allows toggling of cancellation at period end for an active subscription
2020-08-20 17:28:11 +05:30
Daniel Lockyer
04f42732c8 Updated Ghost-Admin to v3.30.1 2020-08-20 09:28:09 +01:00