Commit Graph

8490 Commits

Author SHA1 Message Date
Kevin Ansfield
c1d66f0b01
Added email_recipients include option to members API read endpoint (#12471)
refs https://github.com/TryGhost/Ghost-Admin/pull/1796

We want to be able to display an email activity timeline in Ghost-Admin for each member. The quickest way to achieve that right now is to provide access to the `email_recipient` data for the member when fetching, this will allow clients to build up a timeline based on the event timestamps included with each email_recipient/email pair.

- sets up `email_recipients` relationship in `Member` model
- updates members API read endpoint to accept an `email_recipients` include parameter
  - appends `email_recipients.email` to the `withRelated` array when `email_recipients` is included so that we have data available for email subject and html/plaintext for previews
- updates members API output serializer to include the email_recipients object in the output
2020-12-10 10:04:05 +00:00
Kevin Ansfield
9586d1ce56 Fixed sqlite storing member email open rate as floats
no issue

- sqlite will store a float in an integer column due to it's type affinity resulting in long decimal numbers in the UI when we're expecting an integer
- use the `ROUND()` function to ensure we're storing integers in place of floats when performing open rate average calculations
2020-12-10 10:00:25 +00:00
Fabien O'Carroll
32fe260763 Updated canary members controller to use new Importer
no-issue

This completely replaces the old import functionality with the new
importer!
2020-12-09 19:25:57 +00:00
Fabien O'Carroll
cfad396652 Exported MembersCSVImporter from Members service
no-issue

This will allow the controller to be updated to use the new importer
2020-12-09 19:25:57 +00:00
Fabien O'Carroll
ff7773d1cf Added new members importer module
no-issue

This module encapsulates the work around performing imports, it
currently uses the concept of a "Job" which at the moment is not
persisted to the database, however when we want to look at resuming
imports after a server restart, this should give us the flexibility to
do it.
2020-12-09 19:25:57 +00:00
Fabien O'Carroll
c7359a083f Added email template for completion of background jobs
no-issue

This will be used to generate email content used for notifying users
that their import has been completed.
2020-12-09 19:25:57 +00:00
Fabien O'Carroll
73dc19e498 Removed Members CSV import validation endpoint
no-issue

The new import flow does not use prevalidation of the CSV file, so we
have no use for this anymore.
2020-12-09 19:25:57 +00:00
Fabien O'Carroll
83c5270217 Removed support for batched CSV importer
no-issue

We are rewriting the Members CSV importer to use background jobs, the
batched importer will no longer be used locally.
2020-12-09 19:25:57 +00:00
Kevin Ansfield
baf635e27b Updated email stats aggregator for new member count columns
refs https://github.com/TryGhost/Ghost/issues/12461

- added two default aggregations for overall email count and opened email count
- when number of tracked emails is sufficient add the open rate aggregation to the update query
2020-12-09 13:14:39 +00:00
Kevin Ansfield
b1aafd715d
Added members.email_{count,opened_count} column migrations (#12470)
refs https://github.com/TryGhost/Ghost/issues/12461

- adds `members.email_count` and `members.email_opened_count` columns to contain cached counts for faster queries when outputting member data via API
- adds migration to populate cached counts with existing data
  - tested locally on ~50k members which took ~4sec on mysql
- updates members output serializer to include the new fields in API output
2020-12-09 12:21:56 +00:00
Thibaut Patel
c25344d414
Refactored core/server/lib/image for Dependency Injection (#12463)
no issue
2020-12-09 13:19:22 +01:00
Kevin Ansfield
748a8e0d0f
Added email_open_rate order option to members API (#12462)
refs https://github.com/TryGhost/Ghost/issues/12421

- add `orderRawQuery` function to members model so that we can ensure members with an open rate are ordered before members without an open rate no matter the order direction chosen
- added `email_open_rate` to members in the test fixtures to allow testing of order
2020-12-08 12:43:48 +00:00
Kevin Ansfield
567eb6325f
Added members.email_open_rate aggregation to email analytics (#12458)
refs https://github.com/TryGhost/Ghost/issues/12421
requires https://github.com/TryGhost/Ghost/pull/12457

- updates stats aggregator to calculate and store an open rate for each member
  - uses two queries because I couldn't find a reasonable approach to perform the update in a single query as per the email aggregation
  - benchmarked locally at <1sec/1000members
  - will not store an open rate unless the number of tracked emails sent to a member is above a certain threshold (defaults to 5) to avoid new members being heavily weighted
- fixes typo in EmailAnalytics that was stopping member stats from being aggregated
2020-12-08 12:43:10 +00:00
Kevin Ansfield
9fd6f30fd7
Added migration for members.email_open_rate column (#12457)
refs https://github.com/TryGhost/Ghost/issues/12421

- nullable so we can distinguish between members that have and haven't received any trackable emails
- indexed because we'll be using this column for sorting
2020-12-08 12:20:17 +00:00
Daniel Lockyer
99632eab43
Merged 3.39.3 into master
v3.39.3

* tag '3.39.3':
  v3.39.3
  Updated Ghost-Admin to v3.39.3
2020-12-08 09:42:01 +00:00
Daniel Lockyer
894d18c641
Updated Ghost-Admin to v3.39.3 2020-12-08 09:37:00 +00:00
Daniel Lockyer
12470de19c
Merged 3.39.2 into master
v3.39.2

* tag '3.39.2':
  v3.39.2
  Updated Ghost-Admin to v3.39.2
  Added guard for page.items existing in Mailgun response
2020-12-07 11:17:37 +00:00
Daniel Lockyer
cf744daa51
Updated Ghost-Admin to v3.39.2 2020-12-07 11:14:54 +00:00
Kevin Ansfield
2fe94f34ff
Added guard for page.items existing in Mailgun response
no issue

- it's possible to get Mailgun responses where the `items` array doesn't exist so we need to guard against that so we don't error
2020-12-07 11:12:03 +00:00
Kevin Ansfield
84ae8585c6 Added guard for page.items existing in Mailgun response
no issue

- it's possible to get Mailgun responses where the `items` array doesn't exist so we need to guard against that so we don't error
2020-12-07 11:07:25 +00:00
Daniel Lockyer
f351357159
Merged 3.39.1 into master
v3.39.1

* tag '3.39.1':
  v3.39.1
  Updated Ghost-Admin to v3.39.1
2020-12-07 09:23:12 +00:00
Daniel Lockyer
801729cd7f
Updated Ghost-Admin to v3.39.1 2020-12-07 09:20:09 +00:00
Kevin Ansfield
7c57ffd6a6 Fixed posts without open rate being ordered in reverse
refs https://github.com/TryGhost/Ghost/issues/12420

- adds additional sorting by post creation date so that posts below those with an open rate appear in a more useful order
2020-12-04 13:12:14 +00:00
Thibaut Patel
7038f381b3 🐛 Fixed page preview
issue #12444
2020-12-04 11:35:15 +01:00
Kevin Ansfield
9ff7423b2b
Added email.open_rate order option to posts api (#12439)
refs https://github.com/TryGhost/Ghost/issues/12420

- updated `order` bookshelf plugin's `parseOrderOption()` method to return multiple order-related properties
  - `order` same as before, a key-value object of property-direction
  - `orderRaw` new property that is a raw SQL order string generated from `orderRawQuery()` method in models
  - `eagerLoad` new property that is an array of properties the `eagerLoad` plugin should use to join across
- updated `pagination.fetchAll()` to apply normal order + raw order if both are available and to handle eager loading / joins when `options.eagerLoad` is populated
- updated post model to include details for email relationship and to add `orderRawQuery()` that allows `email.open_rate` to be used as an order option
2020-12-03 20:13:37 +00:00
Daniel Lockyer
47397ca7a6
Updated logging message for webhooks
- removes double space and tidies up the output
2020-12-03 17:50:53 +00:00
Daniel Lockyer
40f9bf3769
Updated Ghost-Admin to v3.39.0 2020-12-02 15:11:32 +00:00
Kevin Ansfield
a945ae7c02 Added sentry error capturing to email analytics jobs
no issue

- jobs operate in their own process and so do not share any of the error capturing that is configured in the parent process
2020-12-02 14:57:15 +00:00
Kevin Ansfield
c40a8e978d Fixed incorrect knex syntax
refs 166d072cd4

- accidentally committed an earlier attempt 🤦🏻‍♂️ updated with the valid syntax
2020-12-02 14:53:10 +00:00
Kevin Ansfield
166d072cd4 Fixed email analytics job not being registered when creating an email
no issue

- job registration was checking for submitted emails in it's email count but the job registration method is called as soon as an email is created meaning the email has a status of 'pending' which prevented the analytics job from being started until a second email was sent
2020-12-02 14:47:18 +00:00
Kevin Ansfield
f802128cfc
Added emailAnalytics config feature flag (#12443)
no issue

- email analytics may be desirable to fully switch off in certain circumstances, when that happens we want to prevent related background jobs from running and expose the feature flag via the config endpoint in the Admin API so that clients can adjust accordingly
2020-12-02 13:22:12 +00:00
Kevin Ansfield
2951eb9eaf Added missing moment require
refs 92ef83c61a
2020-12-02 12:15:27 +00:00
Kevin Ansfield
92ef83c61a Adjusted email jobs registration to check for emails newer than 30 days
no issue

- if emails are older than 30 days we wouldn't be able to fetch any analytics for them and if a site used emails in the past but is no longer using them it doesn't make sense to keep potentially expensive background worker threads spinning up
2020-12-02 12:14:22 +00:00
Daniel Lockyer
947603d0e3
Added original error to thrown error in mega code
- the original error is not propagated up with the error we throw, so it
  can sometimes be difficult to see what's going on down the line
2020-12-02 09:04:20 +00:00
Kevin Ansfield
d675278b0b
Prevented scheduling of recurring analytics jobs when not using emails (#12441)
no issue

- recurring jobs spin up worker threads which can be quite CPU intensive even when not performing much processing, this can be problematic in environments where there are many Ghost instances running
- updated the email job scheduling to be skipped on bootup when there are no emails in the database and to be started when the first email is created as long as we're not in testing env
- increase analytics job schedule from every 2 minutes to every 5 minutes to help spread the load further across instances
2020-12-02 08:17:44 +00:00
Kevin Ansfield
249fd4f06a Fixed email count check in email-analytics service
no issue

- raw knex `.count()` does not return a straight number, we need to handle an array of rowDataPacket objects
2020-12-01 15:02:04 +00:00
Kevin Ansfield
641be8fec3 Adjusted email analytics job scheduling to avoid API spikes
no issue

- typically cron/later schedules will schedule for :00 on the minute which would create API spikes with every members-email-using Ghost site hitting the API at the same time
- adjusted the scheduling to use cron syntax with job runs every 2 minutes on 1,3,5... or 2,4,6... and a random seconds value to smooth usage across sites
2020-12-01 11:27:34 +00:00
Rish
e02ef7f16d Pinned Portal to latest minor version
no refs

Ghost's Portal script is loaded via unpkg which was till now pinned to load `@latest` version, which unpkg auto-resolved to the latest released Portal version. This allowed fast iterations on Portal while still in active beta development to test latest Portal releases.

Going forward, Portal will be pinned to latest specific minor version that allows releasing new features that are not backward compatible without affecting older Ghost releases.

Note: All previous Ghost releases with Portal `@latest` will continue to resolve to latest version and will need to update to latest Ghost 3.x to use all Portal features.
2020-12-01 16:16:14 +05:30
Kevin Ansfield
db7fff0b2a Fixed misleading emailAnalytics.fetchLatest debug statement 2020-12-01 10:16:41 +00:00
Kevin Ansfield
d604f96a9c Added no-emails guard check when fetching latest email events
no issue

- replicates the same guard used in `fetchLatest()` to prevent contacting Mailgun when there are no emails in the database
2020-12-01 10:15:31 +00:00
Kevin Ansfield
b35eef1b02 Added kill-switch config for email analytics background jobs
no issue

- it's possible background jobs may cause unintended side-effects so it's useful to have a kill-switch to disable them individually to keep sites working
2020-12-01 09:10:59 +00:00
Daniel Lockyer
8f86885132 Fixed typo in v2 oembed API
- canary uses `maxwidth` and it's the correct parameter to use
2020-12-01 08:59:41 +00:00
Scott Cabot
da5586e5aa Fixed user import with no created_at date breaking graph
- users imported from CSV with no created_at date where having their created_at date being stored as an int rather than a datetime.
- this was causing parsing issues with the graph so this commit fixes the formatting
2020-11-30 11:21:22 +00:00
Scott Cabot
1e5d201835 🐛 Fixed flat member chart for GMT-X timezones
closes #12083

- fixes a parsing issue where negative offset values were incorrectly having the + sign added regardless of actual offset for sqlite databases.
- for mysql databases absolute values of offset were taken with sign applied where appropriate to stop issues where both hours and minutes could be negative which would cause both an issue with offsets that could present as -2:30 and by the look of the code also trigger extra padding to result in -2:-030 rather than the expected -2:30
2020-11-30 11:21:22 +00:00
Naz
52bcf6086a Fixed malformatted JSDoc {Object} parameters
no issue

- Removed † symbol from the source code
- Not sure how this symbol got into the source code but it definitely should not be here
2020-11-30 17:18:13 +13:00
Naz
eb52cb06bb Improved threading performance on non-SQLite clients
refs https://github.com/TryGhost/Ghost/pull/12431

- child_process are more expensive to run comparing to worker_threads (ref. https://wanago.io/2019/05/20/node-js-typescript-14-performance-hooks/)
- limiting child process only to sqlite backed instances allows high load servers running with MySQL (most usecases) to continue using worker_threads
2020-11-30 13:27:40 +13:00
Kevin Ansfield
a29ac2691a
Fixed sqlite3 errors when email analytics jobs run (#12431)
no issue

- the 4.2.0 version of `sqlite3` that we're using is not compatible with `worker_threads`
- 5.0.0 should add support this but there are other errors
- 5.0.1 is released but not published (https://github.com/mapbox/node-sqlite3/issues/1386)
2020-11-26 15:12:12 +00:00
Daniel Lockyer
40064a395a Switched frontend i18n requires to go through proxy
- we export i18n from `core/frontend/services/proxy` and this is used in
  the most of the places in the frontend code
- this commit aligns the rest of the code in core/frontend to use the
  proxy too
- unfortunately core/frontend/services/themes/i18n.js loops back to the
  proxy so we have a circular dependency
2020-11-26 14:00:28 +00:00
Kevin Ansfield
dcafebe379 Fixed linting 2020-11-26 13:11:18 +00:00
Kevin Ansfield
717543835c
Added email analytics service (#12393)
no issue

- added `EmailAnalyticsService`
  - `.fetchAll()` grabs and processes all available events
  - `.fetchLatest()` grabs and processes all events since the last seen event timestamp
  - `EventProcessor` passed event objects and updates `email_recipients` or `members` records depending on the event being analytics or list hygiene
    - always returns a `EventProcessingResult` instance so that progress can be tracked and merged across individual events, batches (pages of events), and total runs
    - adds email_id and member_id to the returned result where appropriate so that the stats aggregator can limit processing to data that has changed
    - sets `email_recipients.{delivered_at, opened_at, failed_at}` for analytics events
    - sets `members.subscribed = false` for permanent failure/unsubscribed/complained list hygiene events
  - `StatsAggregator` takes an `EventProcessingResult`-like object containing arrays of email ids and member ids on which to aggregate statistics.
  - jobs for `fetch-latest` and `fetch-all` ready for use with the JobsService
- added `initialiseRecurringJobs()` function to Ghost bootup procedure that schedules the email analytics "fetch latest" job to run every minute
2020-11-26 13:09:38 +00:00
Kevin Ansfield
0c59b948fa
Added migrations for email analytics (#12387)
no issue

- cleans up unused tables `emails.{meta,stats}`
- adds timestamp columns `email_recipients.{delivered_at,opened_at,failed_at}` that can be used for event timelines and basic stats aggregation
  - indexed because we want to sort by these columns to find the "latest event" when limiting Mailgun events API requests
- adds aggregated stats columns `emails.{delivered_count,opened_count,failed_count}`
- adds a composite index on `email_recipients.[email_id,member_email]` to dramatically speed up `email_recipient` update queries when processing events
  - modifies the db initialisation to support an `'@@INDEXES@@'` key in table schema definition for composite indexes
2020-11-25 17:48:24 +00:00
Fabien 'egg' O'Carroll
cbaf6e5a74
🐛 Fixed backwards compatibility for newsletters (#12422)
closes https://github.com/TryGhost/Ghost/issues/12416

This fixes compatibility for the `send_email_when_published` option for
the Posts API.

The model layer only allows setting the `email_recipient_filter` column
when the `status` is being changed. Because of this we need to withhold
the `status` change until after we have determined the
`email_recipient_filter`.
2020-11-25 13:58:21 +00:00
Daniel Lockyer
d14f174f35 Refactored 'add column' migrations to use helper
- this removes a lot of code duplication and streamlines how we do
  things across the codebase
2020-11-25 13:56:10 +00:00
Daniel Lockyer
5c3439dc86 Refactored 'add table' migrations to use helper
- this removes a lot of code duplication and streamlines how we do
  things across the codebase
2020-11-25 13:56:10 +00:00
Daniel Lockyer
ed2a21bef4 Added JSDoc comments into package-json lib 2020-11-25 11:29:04 +00:00
Daniel Lockyer
90f7cf9e67 Reverted "Refactored remaining function in package-json lib to use async-await"
- this reverts commit d4b15141a0.
- reverting until I can figure out why the tests are broken
2020-11-25 10:57:55 +00:00
Daniel Lockyer
d4b15141a0 Refactored remaining function in package-json lib to use async-await
- this helps simplify the code and gets rid of Promise chaining
- apparently I can't easily use an async function within filter, so I've
  left it for now
2020-11-25 10:44:15 +00:00
Fabien 'egg' O'Carroll
18b87d9734
Included all subscriptions in stripeSubscriptions (#12414)
refs https://github.com/TryGhost/Ghost/issues/12256

We no longer want to filter out cancelled subscriptions, so we are able
to remove the whereIn clause of the relation.

* Fixed paid flag on member
* Fixed content gating for members

Now that the subscriptions for a member include all of them, we must
explicitly check that the member has an active subscription in order to
consider them "paid"
2020-11-25 10:39:07 +00:00
Daniel Lockyer
ac17b6bc35 Refactored package-json lib to use more async-await
- this helps get rid of all the promise chaining and indentation,
  resulting in cleaner code
2020-11-25 09:19:35 +00:00
Daniel Lockyer
4378435e12 Refactored package-json lib into a class
- this helps bring all the code together so we can extract it in the
  future
- turning it into a class also lets us easily inject the i18n instance
  and store it locally
2020-11-25 09:19:35 +00:00
Kevin Ansfield
47e81e3ca1 Fixed error when creating emails when track opens is disabled
no issue

- ensure `email.track_opens` is a boolean rather than `null`
2020-11-25 08:08:15 +00:00
Naz
0e78f2ed63 Improved job queue logging
no issue

- Makes debugging scheduled jobs easier by listing running/scheduled job names
2020-11-25 13:28:13 +13:00
Fabien 'egg' O'Carroll
217f069085
Included cancellation_reason in subscription serialization (#12411)
refs #12403 

This new property needed to be exposed in serialization so that Ghost-Admin
can use it via the API.
2020-11-23 21:19:27 +00:00
Kevin Ansfield
7bbbeeb5b0 Fixed linting 2020-11-23 19:58:34 +00:00
Kevin Ansfield
a8d8598003 Updated mega service to use open tracking from settings
refs https://github.com/TryGhost/Ghost/issues/12390

- switched from enabling open tracking based on `enableDeveloperExperiments` to using the track-opens setting
2020-11-23 18:53:55 +00:00
Kevin Ansfield
ea77f3f349
Added "email_track_opens" setting (#12404)
refs https://github.com/TryGhost/Ghost/issues/12390

- adds "email_track_opens" setting to the `email` group, defaulting to `true`
2020-11-23 18:36:12 +00:00
Kevin Ansfield
4c96aa5c95 Fixed bulk email only having 'bulk-email' tag in certain circumstances
no issue

- the `'bulk-email`' tag was only being added to bulk emails if another more specific tag was set up via config
- we always want the `'bulk-email'` tag to be present for better event filtering
2020-11-23 18:34:17 +00:00
Daniel Lockyer
15650f9dad Merged 3.38.3 into master
v3.38.3

* tag '3.38.3':
  v3.38.3
  Updated Ghost-Admin to v3.38.3
  Fixed sans-serif link style
2020-11-23 17:37:51 +00:00
Daniel Lockyer
ce3dee0dd4 Updated Ghost-Admin to v3.38.3 2020-11-23 17:34:33 +00:00
Fabien 'egg' O'Carroll
47e266e6d3
Added cancellation_reason column to stripe subscriptions (#12405)
refs #12403 

This will allow us to store the reason for each individual subscription
cancellation.
2020-11-23 15:36:14 +00:00
Peter Zimon
03dae728cb Fixed sans-serif link style
no refs.
- fixed regression that links in emails with sans-serif setting wasn't underlined
2020-11-23 15:28:25 +00:00
Peter Zimon
bf9820f97d Fixed sans-serif link style
no refs.
- fixed regression that links in emails with sans-serif setting wasn't underlined
2020-11-23 15:50:02 +01:00
Fabien 'egg' O'Carroll
f8b617af64
Supported redirecting externally after signup (#12391)
refs #12391

Adds support for redirecting to external URL's after successful signup for members.
2020-11-23 09:36:45 +00:00
Naz
5c4e884070 Added support for graceful job shutdown and worker threads
refs #12402

- With bumped version of job-manager it offloads job procesing into separate worker thread. Having jobs run out of main Ghost process even loop allows for safe job execution, which does not block Ghost from serving requests or performing other functions without a delay
- Added experimental data access to 'testmode' jobs. This should serve as an illustration of how to access data from the job layer
2020-11-23 17:17:49 +13:00
Naz
6ba6e9c307 Removed unused authors input serializer
refs https://github.com/TryGhost/Ghost/issues/10354

- The serializer wasn't hooked up properly during the implementation (1a4497fc9a). It is not possible to hook it back in now as that would introduce breaking change to now stable v2 API.
- v3 (canary) has is properly hooked in, so there should not be a problem going forward
2020-11-23 11:35:02 +13:00
Thibaut Patel
bda4e0da70 Fixed Personal Tokens migration file name
no ref
2020-11-20 10:35:31 +01:00
Thibaut Patel
5fefa9fe96 Disabled Personal Tokens of inactive/locked users
no issue
2020-11-20 09:53:18 +01:00
Thibaut Patel
d48febbbd8 Added Personal Token removal when a user is deleted
no issue
2020-11-20 09:53:18 +01:00
Thibaut Patel
4edccfd2f7 Added the Admin API and tests for Personal Tokens.
no issue
2020-11-20 09:53:18 +01:00
Thibaut Patel
be4146e324 Enabled authentication using an api key with a user_id
no issue

- Enables the Personal Tokens feature.
2020-11-20 09:53:18 +01:00
Thibaut Patel
214ed405cc Added the user_id field to the api_keys table
no issue

- Migration related to the Personal Tokens feature.
2020-11-20 09:53:18 +01:00
Fabien 'egg' O'Carroll
38d19c999d
Removed search params from custom signup redirects (#12386)
refs #12366

This means that custom redirects will not trigger the built in notifications handled by Portal.
2020-11-19 14:47:08 +00:00
Daniel Lockyer
49197a6597 Merged 3.38.2 into master
* origin/3.38.2:
  v3.38.2
  Updated Ghost-Admin to v3.38.2
  🐛 Added missing Bluebird require in v2 API
2020-11-19 13:43:18 +00:00
Daniel Lockyer
20bd729191 Updated Ghost-Admin to v3.38.2 2020-11-19 13:40:08 +00:00
Daniel Lockyer
c72ecf6a96 🐛 Added missing Bluebird require in v2 API
- this was missed in the original pass when removing the global Promise
  override
2020-11-19 13:36:15 +00:00
Fabien 'egg' O'Carroll
9ba3e96790
Redirected to custom path on successful signup (#12372)
refs #12366 

This implements redirection based on the settings for successful member sign up!
- Removes support for redirecting to `req.path` afterwards, this was never used and
  we now have a more configurable implementation.
- Retains redirection to the homepage for unsuccessful sign up (invalid/expired token)
2020-11-19 09:58:32 +00:00
Fabien 'egg' O'Carroll
2600a20f54
Added new settings for members signup redirects (#12370)
refs #12366

Adds two new settings for redirecting members after signup, based on if
they're paid or free, defaulting to the current redirect of "/"
2020-11-17 17:17:54 +00:00
Daniel Lockyer
dccbaf847c Updated Ghost-Admin to v3.38.1 2020-11-17 16:42:06 +00:00
Fabien 'egg' O'Carroll
8ad995203e
🐛 Fixed migration for SQLite3 (#12371)
no-issue

The current version of SQLite3 bundled with Ghost supports a maximum of 999
variables, we use one variable for the SET value and so we're left with 998 for the
WHERE IN clause values.

refs: https://forum.ghost.org/t/unable-to-start-ghost-3-38-0-locally/18322
2020-11-17 16:39:42 +00:00
Daniel Lockyer
99c24ca109 Updated Ghost-Admin to v3.38.0 2020-11-17 13:53:34 +00:00
Fabien 'egg' O'Carroll
bfc152bede
Fixed handling of email_recipient_filter option (#12369)
no-issue

This logic would assume that the option was always passed at the point
of publishing the post, which is not the case for scheduled posts.

Instead of setting the property to 'none' when the option is not
present, we take the approach of ONLY setting the propery when
1. It is present and not 'none'
2. The post is being published or scheduled

This means that scheduled posts will have the property set correctly,
and any future publishing will leave the it in the original state
2020-11-17 11:00:03 +00:00
Rish
f53ab0f52b Updated default config for newsletter badge
no refs

- Migrations were using previous default for badge which was set to `false`
- Default config for badge was `false` which caused migration to switch off the badge, updated to `true`
2020-11-17 16:06:31 +05:30
fueko
8db835a057
Removed unneeded "type='text/css'" in members styles (#12329)
no issue

- Removed unneeded "type='text/css'" attribute from style tag when loading member related styles. 
- Having this attribute makes W3C validation fail.
- Reference on style tag's "type" attribute - https://developer.mozilla.org/en-US/docs/Web/HTML/Element/style#attr-type
2020-11-17 15:28:51 +13:00
Naz
6268d9c324 Refactored UrlGenerator simplifying conditional logic
no issue

- The logic inside the if/esle conditions was duplicated and made it confusing to reason about.
- This change is not effecting the underlying logic at all
2020-11-17 13:09:22 +13:00
Naz
05e12d4b5a Added jsdocs to UrlGenerator/Resource classes
no issue

- The @param {Resource} syntax was not semantically correct. After the change TypeScript is picking up a correct parameter
2020-11-17 13:05:40 +13:00
Daniel Lockyer
33bc3c9011
Refactored column migrations to use helpers (#12360)
- these helpers remove a lot of the duplicated code that we had when doing up/down column migrations, and provides a much shorter way of doing this in the future
2020-11-16 11:55:51 +00:00
Peter Zimon
02d78a89aa Udpated email template footer spacings
no refs.
- updated padding between Ghost badge and rest of the footer
2020-11-16 12:31:37 +01:00
Rish
2d04bfbac9 Enabled newsletter badge by default
no issue

Updated newsletter badge to be shown by default which was previously set to be hidden. Users can still control the visibility via Email settings in Ghost-Admin
2020-11-16 15:55:05 +05:30
Peter Zimon
cfac18cb32 Newsletter email template refinements
no refs.
- updated color styles, links for footer text
- refined spacings for email footer paragraphs
2020-11-13 16:58:56 +01:00
Fabien 'egg' O'Carroll
3f9053ae94
Added setting for newsletter footer content (#12359)
no-issue

We want to give users to ability to customise the content of their newsletter, and the first step
toward that is a setting in which we can store text or html to embed in the template
2020-11-13 13:18:47 +00:00
Fabien 'egg' O'Carroll
4604ba1587
Fixed backward compatibility for send_email_when_published (#12357)
no-issue

* Handled send_email_when_published in Posts API

This restores backwards compatibility of the Posts API allowing existing
clients to continue to use the `send_email_when_published` flag. This
change uses two edits, which is unfortunate. The reason being is that
this is an API compatibility issue, not a model issue, so we shouldn't
introduce code to the model layer to handle it. The visibility property
of the model is used to determine how to fall back, and because it can
be left out of the API request, and relies on a default in the settings,
we require that the model decide on the `visibility` before we run our
fallback logic (or we duplicate the `visibility` default at the cost of
maintenance in the future)

* Dropped send_email_when_published column from posts

Since this column is not used any more, we can drop it from the table.
We include an extra migration to repopulate the column in the event of
a rollback

* Updated importer to handle send_email_when_published

Because we currently export this value from Ghost, we should correctly
import it. This follows the same logic as the migrations for this value.

* Included send_email_when_published in API response

As our v3 API documentation includes `send_email_when_published` we must
retain backward compatibility by calculating the property.

* Fixed fields filter with send_email_when_published

* Added safety checks to frame properties

Some parts of the code pass a manually created "frame" which is missing
lots of properties, so we check for the existence of all of them before
using them.

* Fixed 3.1 migration to include columnDefinition

We require that migrations have all the information they need contained
within them as they run in an unknown state of the codebase, which could
be from the commit they are introduced, to any future commit. In this
case the column definition is removed from the schema in 3.38 and the
migration would fail when run in this version or later.
2020-11-11 13:03:41 +00:00
Peter Zimon
215bfd0a7a
Added newsletter design settings (#12352)
refs https://github.com/TryGhost/Ghost/issues/12355

- Adds new default settings for newsletter customisations - `newsletter_show_badge`, `newsletter_show_header` and `newsletter_body_font_category`
- Adds migrations to update group for new settings
- Add migration to update settings based on existing config value for newsletter settings
- Passes new newsletter settings to newsletter template and updates design based on them
- Fix tests
2020-11-11 18:26:11 +05:30
Daniel Lockyer
f568c4a65e Moved Sentry noop function into block
- this function can be reduced in scope as it's only used in one of the
  branches
2020-11-10 11:04:21 +00:00
Naz
5b6fe31002 Added support for jobs defined as modules
no issue

- These changes should allow easy testing of scheduled jobs and provide an implementation example for future jobs within Ghost
- Added experimental support to run jobs based on functions defined in modules
- Extendeded testmode "schedule" endpoint with an optional job name parameter which picks up job from "jobs" folder
- Adjusted scheduleJob method use and passed "schedule" as first parameter
- Bumped job-manager version to allow for all new functionality
2020-11-10 17:29:35 +13:00
Naz
d6505899a7 Moved testmode routes into their own folder
no issue

- Living in separate folder will allow adding more testmode specific modules (for example test jobs)
- Mimicks the folder structure of how v2/canary routes are structured better
2020-11-10 15:01:28 +13:00
Fabien 'egg' O'Carroll
6140a98351
Updated newsletter functionality to use email_recipient_filter (#12343)
no-issue

* Used email_recipient_filter in MEGA

This officially decouples the newsletter recipients from the post
visibility allowing us to send emails to free members only

* Supported enum for send_email_when_published in model

This allows us to migrate from the previously used boolean to an enum
when we eventually rename the email_recipient_filter column to
send_email_when_published

* Updated the posts API to handle email_recipient_filter

We now no longer rely on the send_email_when_published property to send
newsletters, meaning we can remove the column and start cleaning up the
new columns name

* Handled draft status changes when emails not sent

We want to reset any concept of sending an email when a post is
transition to the draft status, if and only if, and email has not
already been sent. If an email has been sent, we should leave the email
related fields as they were.

* Removed send_email_when_published from add method

This is not supported at the model layer

* Removed email_recipient_filter from v2&Content API

This should not be exposed on previous api versions, or publicly

* Removed reference to send_email_when_published

This allows us to move completely to the email_recipient_filter
property, keeping the code clean and allowing us to delete the
send_email_when_published column in the database. We plan to then
migrate _back_ to the send_email_when_published name at both the
database and api level.
2020-11-06 17:32:23 +00:00
Fabien O'Carroll
44b43a4755 Fixed error with removal of global.Promise override
refs #12182

These were missed in the cleanup and were causing errors.
2020-11-06 16:49:10 +00:00
Fabien O'Carroll
0ce8e3171b Bound SSOAdapter methods to their instance
no-issue

This allows custom SSO adapters to store instance config and refer to
in their methods
2020-11-05 13:08:11 +00:00
Kevin Ansfield
aface9ed4c Enabled Mailgun open tracking when dev experiments is enabled
no issue

- set `emails.track_opens` to `true` when the `enableDeveloperExperiments` flag is set
- update mailgun bulk-email provider to pass the open-tracking header to Mailgun when the email's `track_opens` flag is set
2020-11-05 12:55:08 +00:00
Daniel Lockyer
c8f23873c6 Fixed wording on warning message
- derp
2020-11-05 12:18:28 +00:00
Daniel Lockyer
d8fdf2052c Added check for missing DB data
- this check should never really be hit unless the DB has missing data,
  but the code tries to read from an undefined variable and crashes, which we
  can protect against
- this can be triggered by doing a `knex-migrator init` followed by a
  `knex-migrator rollback --v 3.0 --force`
2020-11-05 12:13:58 +00:00
Daniel Lockyer
a3c8b032de Added missing Bluebird requires
- 504509bb6 removed the global override for Promise
- there are a bunch of places in code that use Bluebird Promise methods,
  but Bluebird wasn't being imported in these places
- this would have thrown errors all over the place
2020-11-05 11:55:14 +00:00
Kevin Ansfield
d41c2f8446
Added migration for emails.track_opens column (#12331)
no issue

- tracking of bulk email opens can be enabled/disabled over time, if we're calculating analytics for emails we don't want emails which didn't have tracking enabled skewing the results so we need a record of whether tracking was enabled for each email
2020-11-05 11:39:53 +00:00
Naz
c7541a1fe0 Added scheduled job support in testmode
no issue

- We need an ability to early test scheduled jobs to refine the new feature and it's API. Should be used with caution
- To schedule an example scheduled job every 30 seconds run following request: `curl http://localhost:2368/ghost/api/schedule/every%2030%20seconds`
2020-11-05 17:46:57 +13:00
Naz
fae05558f3 Added JSDoc comments to redirects service
refs #11085
2020-11-05 12:45:34 +13:00
Naz
f69526c140 Fixed error logged for absent redirects config
refs #11085

- Incorrect usage error was logged to the output when there was no recirecst configuration file present in the system. Previously an empty string was returned in such situation, resulting in "ENOENT" error, which was ignored through special handling.
- The fix resembles logic in redirects async getter function where empty array is returned when the config file does not exits.
- Attempting to read unexistent config should not ever happen and will be handled on the config service layer, this is why special "ENOENT" handling has been removed
2020-11-05 12:45:02 +13:00
Fabien O'Carroll
601b6ce174 Added migration to populate recipient_filter
no-issue

This populates the recipient_filter column for existing emails
based on the visibility of the related post at the time of the
migration.
2020-11-04 16:29:54 +00:00
Fabien O'Carroll
918c721bd1 Added recipient_filter column to emails table
no-issue

This column will allow us to store the canonical recipient filter on the
email resource giving us a detailed log of which members an email was
intended for
2020-11-04 16:29:54 +00:00
Fabien O'Carroll
dd0f80c9fa Added migration to populate email_recipient_filter
no-issue

This populates the email_recipient_filter column for existing posts
based on 1. if they were sent as an email and 2. their visibility.
2020-11-04 16:29:54 +00:00
Fabien O'Carroll
3b0a84b33b Added email_recipient_filter column to posts table
no-issue

This column will allow us to decouple the recipients of newsletter from
the `visibility` of a post, allowing us to send emails to specifically
free members, or to send paid posts as newsletters to all members.
2020-11-04 16:29:54 +00:00
Daniel Lockyer
9b45ea6f7d Added original error to logged error message
- without this, the original error stacktrace is hidden which makes
  issues harder to track down
2020-11-04 14:00:59 +00:00
Kukhyeon Heo
504509bb67
Removed global.Promise override (#12182)
closed #11943 

* Remove global.Promise
* Fix brute-knex bluebird error.
* Fix api-acceptance tests.
* Fix unit tests
2020-11-04 10:55:47 +00:00
Naz
e3e15646eb Updated redirect error message
refs #11085

- The logic which uses this error deals with an input yaml file thus the change.
- Related discussion here - https://github.com/TryGhost/Ghost/pull/12187#discussion_r508144220
2020-11-04 12:26:22 +13:00
Kukhyeon Heo
5715aa2155
Added .yaml format support in redirects configuration (#12187)
closes #11085

- Ghost has been using YAML format for other configurations (e.g. routes). The plan is to move to this format for all user-edited settings files. By default JSON format is still used in Ghost Admin API v2/v3, but will be changed to YAML in API v4. Check referenced issue for more context.
- New format supports all the features available before. The main noticeable change is the structure of config file. It is now grouped by redirect HTTP code instead of specifying `"permanent": true | false` attribute for each config property. Example format for YAML config:
```
302:
  /from-url/: /to-url/

301:
  /category/([a-z0-9\-]+)/i: /tag/$1/
  /v([0-9\.]+)/docs/([a-z0-9\-]+)/i: /docs/$2/
```
- Added 2 new endpoints: `POST redirects/upload` and `GET redirects/download`. These serve as an alias to current GET/POST `/redirects/json. "upload/download" naming pattern is introduced to match the convention with other resources that can be uploaded and downloaded (images, themes etc.). `/redirects/json`  endpoints will be removed in Admin API v4
- The parsing code from `custom-redirects.js` has been moved to `frontend/services/redirects/settings.js`. This location is more appropriate for this logic and eventually `custom-redirects.js` middlewear might be moved into "frontend" as this middlewear plays a role mostly effecting that area.
2020-11-04 12:08:32 +13:00
Daniel Lockyer
300145798e Updated Ghost-Admin to v3.37.1 2020-11-03 14:00:30 +00:00
Daniel Lockyer
4a270632db Updated Ghost-Admin to v3.37.0 2020-11-03 13:11:42 +00:00
Rishabh Garg
8ad11fe082
Enabled Portal (#12317)
no refs

[Portal](https://github.com/TryGhost/Portal) is a new drop-in script to make the bulk of Ghost membership features work on any theme out of the box, which was under a developer flag so far. This release removes the flag for Portal and makes it included as default for any members-enabled Ghost site. The Portal script is backward compatible with old public members script and existing Members-enabled themes should notice no change.

- Removes Portal config flag as Portal is now enabled by default
- Removes old members script as Portal is backward compatible with it
- Changes `{{content}}` helper to show default CTA in case of restricted content access
- `accent_color` setting is no more behind the dev experiment flag and included by default
- Adds migration to switch off Portal button setting for all existing sites which don't have Portal enabled in beta
2020-11-03 14:36:21 +05:30
Naz
4ebebd12d3 Removed unneeded parameter
no issue

- There is no second parameter in Promise constructor. Didn't serve any purpose as per comment - bbbd011074 (r41413465)
2020-11-03 13:38:08 +13:00
Naz
7326241718 🐛 Fixed server error for repeated order query parameter
closes #12263

- Express parses repeated query parameters as an array (req.query properties). Because there is no clear reason on why not to support this behavior extended order parameter parsing logic to handle arrays. This follows the rule of "liberal inputs, conservative outputs"
- Example supported query string for ordering can now look like: `?order=featured&order=published_at asc`, the priority of the order stays the same with the most significant appearing first and least significant last
2020-11-03 00:15:24 +13:00
naz
a2bfef53de
🐛 Fixed post resource filtering by posts_meta table fields (#12307)
refs #11572

- Filtering by fields coming from posts_meta table did not work for post resources. This was due to lack of support for these types of operations on NQL layer. The approach taken here is using same way filtering was done for many:many relations and generates a `WHERE IN` filtering clause. In the future we could look into adding preloading of 1:1 relations which should allow getting rid of `WHERE IN` in favor of `JOIN` and filtering directly by field names. 
- Changed structure of `EXPANSIONS` filter configuration. Current approach was based on "bag of all the things". Such structure will become problematic as more fields are added. For example, adding all the fields from 1:1 relation posts:posts_meta might collide with any other relations that would have similar naming like meta_description from tags table (if it were was added).
- Bumped nql version to 0.5.0. This adds filtering support to 1:1 relations
- Added filter expansions which can be unique per model Previous approach with single global expansions lookup wasn't working in case different models would need to declare expansion for same field names. Having a `filterExpansion` method per model works in a similar convention other filter related model methods do (e.g. enforcedFilters, defaultFilters)
2020-11-02 12:53:57 +13:00
Kevin Ansfield
7a74b78940 Included email-id as user variable when sending bulk emails via mailgun
no issue

- adding user variables via the mailgun API when sending emails means that events related to email have those variables attached to them
- adding the `email.id` value to user variables means we can easily associate mailgun events to emails, otherwise we'd have look up the batch via `email_batches.provider_id` then use a join to get back to the associated email
2020-10-29 16:37:42 +00:00
Daniel Lockyer
85da9bdeb2 Updated Ghost versioning lib to include all prerelease IDs
- currently if Ghost has a version of something like
  `3.37.0-pre.abc.def`, Ghost will return `3.37.0-pre.abc` as the full
  version
- this hides parts of the version which are useful for debugging
- this commit updates the logic to join together all prerelease elements
  so we keep the full string
2020-10-29 12:44:55 +00:00
Rish
c4c6bdf353 Updated mail template for member email change
no issue

The email change verification template was using the same as for `subscribe`, which did not have the right messaging that needs to be communicated about the action thats happening in this case. Updates the email template to same as what we use for email verification for support/newsletter address
2020-10-29 17:40:19 +05:30
Rish
746ccbd7ed Added custom action param for magic links
closes https://github.com/TryGhost/Ghost/issues/12253

- Allows using custom action param for requests from Portal by using a new `requestSrc` option that is passed down when a request for magic link is made via Portal
2020-10-29 13:42:52 +05:30
Rishabh Garg
0ce5f87616
🐛 Added past_due and unpaid subscriptions for members (#12301)
refs https://github.com/TryGhost/Ghost/issues/12256 , https://github.com/TryGhost/Ghost/issues/12255

Currently when listing subscriptions for Members in both the Admin and the Theme, we only show the subscriptions which have a status of trialing or active.

Based on discussion, the `unpaid` and `past_due` states on Stripe also represent owner's intention of considering a subscription as active instead of `cancelled`, so we allow any subscriptions under these 2 states to be also listed for a member and consider them as `paid`.

- Subscriptions will go into a past_due state if the payment is missed, this should be considered a grace period where the member still has access.

- After this the subscriptions will either go to the unpaid or the cancelled state - this can be configured on an account by account basis in the Stripe dashboard. `unpaid` is considered as an intention to keep the subscription to allow for re-activation later.
2020-10-27 15:19:48 +05:30
Peter Zimon
43b7c7b344
Updated members admin redirect to new page(#12296)
no refs.

- Members settings were moved to a new page [here](https://github.com/TryGhost/Ghost-Admin/pull/1736)
- updated members related redirect for from/newsletter email address change to point to new page
2020-10-22 16:10:25 +05:30
Daniel Lockyer
c68c749026 Updated Ghost-Admin to v3.36.0 2020-10-20 09:38:09 +01:00
Rishabh Garg
7182efdb88
🐛 Fixed special chars in single use token (#12290)
no refs

- The token generation logic for single use token was replacing only the first instance of + or / to make the token URL safe, instead of replacing all instances which caused a bug where token was not validated properly in case it included multiple + or / in it.

- The fix ensures replacing all the + or / in the token with URL safe _ or - so it can be properly validated via magic link
2020-10-20 11:49:20 +05:30
naz
8ddf83f3c5
Fixed "no-shadow" linting error in server modules (#12287)
refs 143921948d

- Continuation of changes started in referenced commit
2020-10-20 12:02:56 +13:00
naz
38b7f8e311
Fixed "no-shadow" linting error in server/data modules (#12288)
refs 143921948d

- Continuation of changes started in referenced commit
2020-10-20 11:56:46 +13:00
Kevin Ansfield
d08b586781 🐛 Fixed width/height attributes in gallery cards not matching resized images
closes https://github.com/TryGhost/Ghost/issues/11548

- bumps @tryghost/kg-default-cards
  - introduces a breaking change where `imageOptimization` is passed as an options argument instead of separating out individual config options
  - fixes width/height attributes for images in gallery cards being larger than the max default width used when resizing images
2020-10-19 17:05:57 +01:00
Kevin Ansfield
3148d6a946
Moved definition of default max image width to config (#12289)
no issue

- centralises definition of max width and allows customisation if needed
- allows for passing of the config value through to rendering libraries
2020-10-19 08:56:18 +01:00
Kevin Ansfield
16bfb3fa41 Added basic CRUD admin API for snippets
no issue

- standard browse/read/add/edit/destroy API endpoints for snippets resource
- updates `@tryghost/admin-api-schema` dependency to version that includes snippet definition and schemas
2020-10-16 18:02:58 +01:00
Kevin Ansfield
e5a92be932 Fixed typo in snippets permission fixtures
refs https://github.com/TryGhost/Ghost/pull/12283

- `delete` -> `destroy` for action type
2020-10-16 17:46:31 +01:00
Kevin Ansfield
15afed4b81
Added snippets permissions migration and fixtures (#12283)
no issue

- all staff users can browse/read snippets so they're usable in the editor for everyone
- only administrators, editors, and admin integrations are able to create/edit/delete snippets
2020-10-16 12:53:09 +01:00
Kevin Ansfield
74269070dd
Added migration for snippets table creation (#12282)
no issue

- minimal table structure required for the first iteration of content snippets
- snippets are stored pieces of re-usable content that could effectively be entire posts so the `mobiledoc` field length matches the `posts.mobiledoc` field length
2020-10-16 12:52:06 +01:00
Rish
a5a7515fe8 Fixed typo in private setting import
no refs

Last commit introduced a small typo in how private setting is ignored during import
2020-10-15 15:55:24 +05:30
Rish
3e0948d04e Ignored member email settings in import
no issue

- Member email settings - newsletter and support - was previously being set to null for all cases, which can lead to `null` values being set for those addresses on a successful import whereas the expected behavior is to retain the old values.
- Adds the 2 email settings to ignored settings list so they are not changed during import process
2020-10-15 15:32:05 +05:30
Rish
54e638310c Added default support address for member site data
no issue

- In a case where support address is null, member's site data won't load as there was no fallback.
- The fix adds default `noreply` as the support address for site data
2020-10-15 14:44:16 +05:30
Fabien O'Carroll
a9b3d83e00 Loaded Action model after all other models
no-issue

The Action model loops through all registered models when the file is
loaded, by loading the model last, we ensure that it can read all
models, rather than an arbitrary selection which come before it.
2020-10-14 15:11:45 +01:00
Fabien O'Carroll
0fd0527da0 Fixed creation of EmailBatch and EmailRecipient collections
no-issue
2020-10-14 15:11:45 +01:00
naz
392140cb36
Allowed for comped field when creating a member through Members API (#12278)
closes #12273

- `comped` field has been allowed when editing a member or importing from a CSV. There has been a usecase (Zapier Integration) for API client to create a member with "Complimentary" plan, which made this change necessary
- Previously the logic for comped field was to skip and continue member record creation if Stripe was not connected. Now we throw an error - same as the one we have been throwing before when stripe_customer_id field was passed in. The implication of this change is that we won't be creating any record now if comped === true and Stripe is disabled. 
- Bumped admin-api-schema-package. Contains `comped` schema change so this field gets passed through to controller
2020-10-14 13:24:09 +13:00
Daniel Lockyer
39ff4fe688 Updated Ghost-Admin to v3.35.5 2020-10-08 11:00:35 +01:00
Kevin Ansfield
8ad384d7f8 🐛 Fixed scheduled post emails pointing at /404/ for the "view online" link
no issue

When scheduling a post to publish+send the "view online" link was pointing at https://site.com/404/ rather than the published post's url.

The problem occurred because the `/schedules/` endpoint wraps it's post read+edit calls in a transaction. Context:
- when a post is published with with the "send email" option the email record is immediately generated and added to the API response, as part of the email record generation we render the email content including fetching the url for the "view online" link
- urls for all resources are handled by our `url` service, that service updates it's internal cache based upon model events such as the "edited" event triggered when a post is published
- if the posts API controller is given a transaction, the email record is also generated inside of that transaction however at this point the `url` service will not have been updated because the post record hasn't been committed meaning it has no available url for the post

Fix:
- removed the `models.Base.transaction()` wrapper around the post read+update in the `/schedules/` API controllers
- we don't need a transaction here. It was added as protection against another write request coming in between the `/schedules/` controller reading a post and publishing a post but we already have protection against that in the form of collision detection - if a write request comes in and commits between the schedules controller reading the post and updating it, the scheduler's update call will fail with a collision error at which point the scheduler itself should retry the request which could then publish the post successfully if everything else is in order
2020-10-08 10:26:49 +01:00
Daniel Lockyer
90a00ab3e5 Updated Ghost-Admin to v3.35.4 2020-10-06 09:55:49 +01:00
Kevin Ansfield
9cbeb74db0
🐛 Fixed broken embeds cards when pasting links to Wordpress sites (#12262)
closes https://github.com/TryGhost/Ghost/issues/12260

- if a card type was not explicitly chosen (i.e. a url was pasted into the editor) then abort fetching the oembed endpoint if we detect it's a `wp-json` oembed and return a bookmark card payload instead
- cleaned up an unused argument in the internal `fetchBookmarkData()` method
2020-10-06 08:44:03 +01:00
Kevin Ansfield
cde364bf27 🐛 Fixed email card replacements showing raw replacement text in emails
closes https://github.com/TryGhost/Ghost/issues/12257

- there was a destructuring problem introduced in the recent email refactor which meant the correct replacement data was not being passed over to the Mailgun provider when sending email
2020-10-05 17:24:48 +01:00
Kevin Ansfield
4f211d025d Fixed members with multiple subscriptions receiving multiple newsletters
closes https://github.com/TryGhost/Ghost/issues/12259

- adds a `DISTINCT` to the query used to fetch member rows when generating an email recipient list
- this increases query time 2.7s vs 1.6s locally with ~94k paid members but once the `members.paid` column is implemented this slow query can be removed
2020-10-05 16:53:35 +01:00
Naz
39e403d176 🐛 Fixed meta attributes calculation on post preview
closes https://github.com/TryGhost/Ghost/issues/12247

- Internal preview controller was lacking "mapping" call to post object which handled not only missing meta attribute information but lots of other mappings (e.g. users, tags, etc.)
- Have added a regression test to catch issues like this in the future
2020-10-05 17:52:40 +13:00
Sam Lord
46c20759f3 Updated Ghost-Admin to v3.35.3 2020-10-02 14:38:32 +01:00
Kevin Ansfield
8f3ab3c535 🐛 Fixed email showing as success when an email batch fails to send
no issue

- fixed passing of errors up through send/processBatch/processEmail
- fixed errant overwrite of email status with a "submitted" status after a failure had occurred
2020-10-02 14:26:57 +01:00
Kevin Ansfield
b5ffb38726 Fixed partial email batch/recipient records when email creation fails
no issue

- wrap email batch/recipient record creation in a transaction so if an error occurs during creation we're not left with a partially created batch/recipient set in the database
2020-10-02 13:47:14 +01:00
Kevin Ansfield
0f30b9f6a0 🐛 Fixed email not showing as failed if error occurs when preparing email
no issue

- if an error occurred whilst creating email batch/recipient records the email status was never updated and was left in the 'pending' status
- adjusted the error handling to update the email status and record the error message if such a scenario occurs
2020-10-02 12:40:49 +01:00
Kevin Ansfield
7b789e1cd5 🐛 Fixed newsletters being sent to Stripe customer emails in place of member emails
no issue

- the paid-member SQL query that is obtained using `models.Member.getFilteredCollectionQuery({paid: true})` can return multiple columns with the same name (eg, `email`, `name`), when that happens the last column with duplicate names "wins" and it's value is used in the resulting knex row instance
- in the `mega` service when fetching email recipient rows we ran into this problem, to avoid it we adjust the query to explicitly select only the data from the `members` table
2020-10-02 12:17:17 +01:00
Kevin Ansfield
82126f29e6 Added guard for member rows having missing data when creating email recipients
no issue

- we've had an issue with emails failing due to unexpectedly missing data when inserting email recipient rows
- added a validation check before adding recipient details along with a log so that invalid data can be investigated
2020-10-02 11:29:10 +01:00
Rish
58fda5bad2 Allowed accent color setting for site with portal flag
no issue

- We had previously allowed accent_color setting for member site settings behind portal flag, but Ghost Admin also needs the public site setting with accent color to correctly reflect the accent color when flag is switched on
- Removes deletion of accent color setting when behind the Portal flag OR dev experiment flag
2020-10-02 15:30:54 +05:30
Sam Lord
d812e57813 Updated Ghost-Admin to v3.35.2 2020-10-01 19:57:14 +01:00
Kevin Ansfield
defb43fe7a 🐛 Fixed newsletters emails having no subject
no issue

- subject was not being picked out of the message data when passing over to the mailgun-js send method
2020-10-01 19:51:01 +01:00
Sam Lord
3e0f8c1eb9 Updated Ghost-Admin to v3.35.1 2020-10-01 19:21:26 +01:00
Kevin Ansfield
e34acc31c5 🐛 Fixed newsletter email sending
no issue

- there was an typo in the recent email sending refactor that resulted in `Error: 'to' parameter is missing` errors when sending email previews and bulk emails
2020-10-01 19:03:57 +01:00
Fabien 'egg' O'Carroll
ce14d8113a
🐛 Fixed members form missing data-members-name (#12250)
refs #12249

This was incorrectly assuming the presence of the data-members-name
element in the document. By guarding against it and defaulting to
undefined, we fallback to the existing behaviour when the element is not
present.
2020-10-01 18:05:23 +01:00
Fabien 'egg' O'Carroll
244704156c
Updated all Origin header checks to handle 'null' (#12246)
closes #12244

As per RFC 6454 the Origin header MUST be set to the string 'null' when
in a "privacy-sensitive" context. We were not handling this string and
this was causing errors. This commit updates all checks of the 'Origin'
header to treat the value 'null' as if the header was not present.

ref: https://tools.ietf.org/html/rfc6454#section-7.3
2020-10-01 09:37:22 +01:00
Rishabh Garg
eb4933d27e
Added extra settings to member auth emails (#12238)
no issue

- Adds accent color, site domain and url to member auth emails for customization
2020-10-01 12:31:36 +05:30
Sam Lord
4ad4892d3d Updated Ghost-Admin to v3.35.0 2020-09-30 11:00:14 +01:00
Fabien O'Carroll
fd5194ea46 Updated magic link expiry to 24 hours
no-issue

As discussed with @JohnONolan 24 hours is the preferred expiry for magic links
2020-09-30 10:53:35 +01:00
Fabien 'egg' O'Carroll
c19d282a51
Ensured Admin API members resource only returns known fields (#12240)
refs #12055

As part of the work in TryGhost/Members#206 we load the stripeCustomers relation on the member model, and we do not want this to be part of the API response. The changes here include a refactor but the main thing is that the serialized object is explicit and does not include unexpected or unknown fields.

* Moved mapMember out of mapper file

This cleans up the serializer a bit by keeping it's functionality all in
one place, rather than a shared mapper file

* Refactored members controller to return models

Previously the controller was calling toJSON, which is serialization,
this updates the controller to only deal with models, leaving all of the
serialization to the serializer!

* Refactored members serializer

This adds typings to all of the methods/functions in the serializer, as
well as making the serializating explicit, rather than returning the
result of toJSON, we explicitly set the properties we expect to be on
the output object. This protects us against accidental API changes in
the future.
2020-09-30 10:22:22 +01:00
naz
071bf8edb5
Extracted Admin API JSON schemas into @tryghost/admin-api-schema (#12237)
closes https://github.com/TryGhost/Ghost/issues/10628

- JSON Schemas were extracted into a separate module to allow other clients to reuse them (for example documentation). Having them in a separate package also slims down the amount of code needed to be maintained in the core.
- Updated canary API input validators to use admin-api-schema module
- Removed canary schemas that moved into admin-api-schema package
- Updated v2 API input validators to use admin-api-schema package
- Removed v2 schemas that moved into admin-api-schema package
- Updated tests to contain needed information in apiConfig to pick up correct validation
- Added @tryghost/admin-api-schema package dependency
2020-09-30 16:25:12 +13:00
Kevin Ansfield
932def7e6d Fixed replacement strings showing when viewing sent emails
refs https://github.com/TryGhost/Ghost/pull/12232

When viewing sent emails in Ghost's admin area, it displays the `html` field directly from the `email` relation loaded with the post. Since the `mega` refactor we now store raw content in that field rather than sanitized "preview" content so it's necessary to modify the API output to match the old behaviour.

- use the API output serializers to parse replacements in email content and replace with the desired fallback or empty string
2020-09-29 17:39:22 +01:00
Kevin Ansfield
474e6c4c45 Refactor mega service to use stored email content and batch/recipient records
no issue

- store raw content in email record
  - keep any replacement strings in the html/plaintext content so that it can be used when sending email rather than needing to re-serialize the post content which may have changed
- split post email serializer into separate serialization and replacement parsing functions
  - serialization now returns any email content that is derived from the post content (subject/html/plaintext) rather than post content plus replacements
  - `parseReplacements` has been split out so that it can be run against email content rather than a post, this allows mega and the email preview service to work with the stored email content
- move mailgun-specific functionality into the mailgun provider
  - previously mailgun-specific behaviour was spread across the post email serializer, mega, and bulk-email service
  - the per-batch `send` functionality was moved from the `bulk-email` service to the mailgun provider and updated to take email content, recipient info, and replacement info so that all mailgun-specific headers and replacement formatting can be handled in one place
  - exposes the `BATCH_SIZE` constant because batch sizes are limited to what the provider allows
- `bulk-email` service split into three methods
  - `send` responsible for taking email content and recipients, parsing replacement info from the email content and using that to collate a recipient data object, and finally coordinating the send via the mailgun provider. Usable directly for use-cases such as test emails
  - `processEmail` takes an email ID, loads it and coordinates sending related batches concurrently
  - `processEmailBatch` takes an email_batch ID, loads it along with associated email_recipient records and passes the data through to the `send` function, updating the batch status as it's processed
  - `processEmail` and `processEmailBatch` take IDs rather than objects ready for future use by job-queues, it's best to keep job parameters as minimal as possible
- refactored `mega` service
  - modified `getEmailData` to collate email content (from/reply-to/subject/html/plaintext) rather than being responsible for dealing with replacements and mailgun-specific replacement formats
    - used for generating email content before storing in the email table, and when sending test emails
    - from/reply-to calculation moved out of the post-email-serializer into mega and extracted into separate functions used by `getEmailData`
  - `sendTestEmail` updated to generate `EmailRecipient`-like objects for each email address so that appropriate data can be supplied to the updated `bulk-email.send` method
  - `sendEmailJob` updated to create `email_batches` and associated `email_recipients` records then hand over processing to the `bulk-email` service
  - member row fetching extracted into a separate function and used by `createEmailBatches`
  - moved updating of email status from `mega` to the `bulk-email` service, keeps concept of Successful/FailedBatch internal to the `bulk-email` service
2020-09-29 17:17:54 +01:00
Kevin Ansfield
d34a3263e8 Store email batch and recipient records when sending newsletters (#12195)
requires https://github.com/TryGhost/Ghost/pull/12192

- added initial `EmailBatch` and `EmailRecipient` model definitions with defaults and relationships
- added missing `post` relationship function to email model
- fetch member list without bookshelf
    - bookshelf can add around 3x overhead when fetching the members list for an email
    - we don't need full members at this point, only having the data is fine
    - if we need full models later on we can push the model hydration into background jobs where recipient batches are fetched ready for an email to be sent
    - bookshelf model instantiation of many models blocks the event loop, using knex directly keeps concurrent requests fast
    - adds `getFilteredCollectionQuery` method to base model to facilitate getting a knex query based on our normal model filters along with transaction/forUpdate applied
- store recipient list before sending email
    - chunk already-fetched members list into batches and insert records into the `email_recipients` table via knex
    - chunked into batches of 1000 to match the number of emails that Mailgun accepts in a single API request but this may not be the absolute fastest batch size for recipient insertion:
        | Batch size | Batch time | Total time |
        | ---------- | ---------- | ---------- |
        |        500 |       20ms |     4142ms |
        |       1000 |       50ms |     4651ms |
        |       5000 |      170ms |     3540ms |
        |      10000 |      370ms |     3684ms |
    - create an email_batch record before inserting recipient rows so we can effeciently fetch recipients by batch and store the overall batch status
2020-09-29 17:17:54 +01:00
Rish
8e0976fc8f Allowed accent color in members site data
no issue

- The accent_color setting was being removed from members site data when behind portal flag
- The accent color setting is now allowed in members site data for all cases as it doesn't make any sense to remove it specifically from here where we already have all the other Portal settings included which is a dev/portal flag feature anyways
2020-09-29 17:23:33 +05:30
Fabien 'egg' O'Carroll
854a41e556
🐛 Fixed CSV import json-schema email validation (#12239)
no-issue

By using the "email" validation, we were validating emails in CSV
imports using a different validator to the rest of the API. AJV's built
in email validation was failing on emails with "special" characters,
such as letters with an umlaut above them.

This commit brings the validation for CSV imports in line with the rest
of the API.
2020-09-29 12:51:35 +01:00
Peter Zimon
4e45373730 Updated Webhook API error message
refs 6f1abc610a

- updated missing Webhook API error message when integration is missing
2020-09-28 13:24:47 +02:00
Naz
50f30c4f66 Fixed minor formatting in i18n file 2020-09-28 14:21:29 +13:00
Daniel Lockyer
685e2782b7 Merged 3.34.1 into master
v3.34.1

* tag '3.34.1':
  v3.34.1
  Updated Ghost-Admin to v3.34.1
  🐛 Fixed email verification mails not sent
2020-09-24 19:28:22 +01:00
Daniel Lockyer
4cb228fab0 Updated Ghost-Admin to v3.34.1 2020-09-24 19:26:52 +01:00
Rish
82399393b8 🐛 Fixed email verification mails not sent
no issue

- Email ownership verification emails for support/from address was not using the updated magic link service syntax
2020-09-24 19:10:57 +01:00
Rish
03968219a7 🐛 Fixed email verification mails not sent
no issue

- Email ownership verification emails for support/from address was not using the updated magic link service syntax
2020-09-24 23:39:25 +05:30
Kevin Ansfield
1dfaf8c5f8
Added migration to store from/reply-to values in email table (#12204)
no issue

The email table should be a reference for all data that was used when sending an email. From and Reply-to addresses can change over time and we don't have any other reference for their value at the time of sending an email so we should store them alongside the email content.

- schema updated with `from` and `reply_to` columns
- both are set to `nullable` because we don't have historic data (can be populated and changed in later migrations if needed)
- neither `from` or `reply_to` have `isEmail` validations because they can have name+email in an email-specific format
- will help keep concerns separated in the future. `mega` service can deal with all of the email contents/properties, and the `bulk-email` service's concerns are then only email sending and any provider-specific needs
2020-09-24 08:20:10 +01:00
Rish
78379e76c1 Renamed membersjs script name to portal
no issue

- The members.js package was renamed as `@tryghost/portal`, which also updated the unpkg link for the script
- Updates the unpkg script for portal to use the new package name and path
2020-09-24 12:08:51 +05:30
Naz
43153ba31e Disabled changing webhook's integration_id through PUT API
refs #12033

- Allowing to change parent integration opens up possible security holes and has no clear usecase at the moment. After a webhook record is created it should not be possible to change parent integration.
- Had do partially duplicate JSON schema definition from webhooks definition as there is no proper composition technique available in current version of JSON Schema.
2020-09-24 16:24:02 +12:00
Naz
6f1abc610a Added check for parent integration_id when creating a webhook
refs https://github.com/TryGhost/Ghost/issues/12033
refs https://github.com/TryGhost/Ghost/issues/10567

- Creating a webhook without valid parent integration leads to orphaned webhook records, which shoult not ever happen
- This scenario is only possible for non-integration authentication,
because in case of integration being authenticated it's id is
automatically assigned to creatd webhook
2020-09-24 15:09:51 +12:00
naz
bbcc83dadb
Added support for ordering Post API resources by fields coming form posts_meta table (#12226)
refs #11729

- When ordering is done by fields from a relation (like post's `meta_title` that comes form `posts_meta` table), Bookshelf does not include those relations in the original query which caused errors. To support this usecase added a mechanism to detect fields from a relation and load those relations into query. 
- Extended ordering to include table name in ordered field name.  The information about the table name is needed to avoid using `tableName` within pagination plugin and gives path to having other than original table ordering fields (e.g. order by posts_meta table fields)
- Added test case to check ordering on posts_meta fields
- Added support for "eager loading" relations. Allows to extend query builder object with joins to related tables,
which could be used in ordering (possibly in filtering later). Bookshelf does not support ordering/filtering by proprieties coming from relations, that's why this kind of plugin and query expansion is needed
- Added note about lack of support for child relations with same property names.
2020-09-24 13:32:40 +12:00
Kukhyeon Heo
635d26469f
Fixed settings reinit to only emit changed settings events (#12184)
closes #12038

Previously we were emitting changed events for _all_ settings which would
cause any listeners for those to be triggered, this ensures that listeners are
only triggered if the corresponding setting, _did_ in fact change.
2020-09-23 14:35:03 +01:00
Fabien O'Carroll
171908cefa Added support for filtering Members CSV exports
no-issue

Adding these options to the endpoint will pass them to the model layer,
allowing us to filter members CSV exports
2020-09-23 14:29:15 +01:00
Léo Bourrel
87855075c0
Added support for data-members-name in themes (#12191)
no-issue

This change allows themes to collect member names on signup with use of the
`data-members-name` attribute on an input element in the `data-members-form`
2020-09-23 13:59:26 +01:00
Naz
bff2fd5f3b Corrected comment describing redirect behavior
no issue

- The redirect is done from `/feed` to `/rss`
2020-09-23 00:37:55 +12:00
Daniel Lockyer
40cb03d9d1 Updated Ghost-Admin to v3.34.0 2020-09-22 11:25:12 +01:00
Nazar Gargol
84c8bcc457 Extracted ordering into separate bookshelf plugin
refs #11729

- Having a plugin allows to reason about ordering better and gives way
to further extraction away form the core.
- Just like with filtering, ordering falls into similar category of having effect on knex'es query builder object extension through additional statements - ORDER BY in this case.
- Because there were bugs associated with use of permittedAttributes inside of `parseOrderOption` method, introduced separate `orderAttributes` function which returns only those fields which are orderable (https://github.com/TryGhost/Ghost/issues/11729#issuecomment-685740836)
2020-09-22 16:27:12 +12:00
Nazar Gargol
32b37d7ba8 Improved error messaging for password reset process
refs #11878

- When password reset link is invalid previous messaging left the user
without clear information about why the reset failed and what they could do about it.
- Updated messaging around password reset tokens including detection of
when password token has invalid structure, has expired or has already
been used
2020-09-22 15:45:19 +12:00