Commit Graph

12062 Commits

Author SHA1 Message Date
Kevin Ansfield
567eb6325f
Added members.email_open_rate aggregation to email analytics (#12458)
refs https://github.com/TryGhost/Ghost/issues/12421
requires https://github.com/TryGhost/Ghost/pull/12457

- updates stats aggregator to calculate and store an open rate for each member
  - uses two queries because I couldn't find a reasonable approach to perform the update in a single query as per the email aggregation
  - benchmarked locally at <1sec/1000members
  - will not store an open rate unless the number of tracked emails sent to a member is above a certain threshold (defaults to 5) to avoid new members being heavily weighted
- fixes typo in EmailAnalytics that was stopping member stats from being aggregated
2020-12-08 12:43:10 +00:00
Kevin Ansfield
9fd6f30fd7
Added migration for members.email_open_rate column (#12457)
refs https://github.com/TryGhost/Ghost/issues/12421

- nullable so we can distinguish between members that have and haven't received any trackable emails
- indexed because we'll be using this column for sorting
2020-12-08 12:20:17 +00:00
Daniel Lockyer
99632eab43
Merged 3.39.3 into master
v3.39.3

* tag '3.39.3':
  v3.39.3
  Updated Ghost-Admin to v3.39.3
2020-12-08 09:42:01 +00:00
Daniel Lockyer
6ca434a16f
v3.39.3 2020-12-08 09:37:00 +00:00
Daniel Lockyer
894d18c641
Updated Ghost-Admin to v3.39.3 2020-12-08 09:37:00 +00:00
renovate[bot]
85f94ad4b4
Update dependency @tryghost/job-manager to v0.4.0 (#12460)
Co-authored-by: Renovate Bot <bot@renovateapp.com>
2020-12-08 17:48:13 +13:00
Kevin Ansfield
5d11475f32 Fixed false positive image-size tests
no issue

- fixed usage of rewire and sinon spies
  - spies actually set (and reset) on the rewired module so the spies are called in place of the original functions
- fixed should syntax for spies, they were written with chai syntax but we don't use chai in server-side code
- chained `.catch(done)` to the `.catch(assertions)` in tests that are testing errors so that we don't have unhandled rejections and 60s timeouts when an assert fails
- changed `.gif` to `.ico` to match current fetch-only extension list
- fixed "[failure] can handle an error with statuscode not 200 (image-size)" which was expecting the `sizeOf` function to be called when it shouldn't be called
2020-12-07 22:38:14 +00:00
Renovate Bot
3aa43eb36f
Update dependency js-yaml to v3.14.1 2020-12-07 19:50:35 +00:00
Renovate Bot
81d5fb60d1
Update dependency @sentry/node to v5.29.0 2020-12-07 11:18:56 +00:00
Daniel Lockyer
12470de19c
Merged 3.39.2 into master
v3.39.2

* tag '3.39.2':
  v3.39.2
  Updated Ghost-Admin to v3.39.2
  Added guard for page.items existing in Mailgun response
2020-12-07 11:17:37 +00:00
Daniel Lockyer
3d61603719
v3.39.2 2020-12-07 11:14:54 +00:00
Daniel Lockyer
cf744daa51
Updated Ghost-Admin to v3.39.2 2020-12-07 11:14:54 +00:00
Kevin Ansfield
2fe94f34ff
Added guard for page.items existing in Mailgun response
no issue

- it's possible to get Mailgun responses where the `items` array doesn't exist so we need to guard against that so we don't error
2020-12-07 11:12:03 +00:00
Kevin Ansfield
84ae8585c6 Added guard for page.items existing in Mailgun response
no issue

- it's possible to get Mailgun responses where the `items` array doesn't exist so we need to guard against that so we don't error
2020-12-07 11:07:25 +00:00
Daniel Lockyer
f351357159
Merged 3.39.1 into master
v3.39.1

* tag '3.39.1':
  v3.39.1
  Updated Ghost-Admin to v3.39.1
2020-12-07 09:23:12 +00:00
Daniel Lockyer
0f6f5d09db
v3.39.1 2020-12-07 09:20:09 +00:00
Daniel Lockyer
801729cd7f
Updated Ghost-Admin to v3.39.1 2020-12-07 09:20:09 +00:00
Renovate Bot
52b0f970e9
Update dependency eslint to v7.15.0 2020-12-07 00:07:07 +00:00
Kevin Ansfield
7c57ffd6a6 Fixed posts without open rate being ordered in reverse
refs https://github.com/TryGhost/Ghost/issues/12420

- adds additional sorting by post creation date so that posts below those with an open rate appear in a more useful order
2020-12-04 13:12:14 +00:00
Thibaut Patel
7038f381b3 🐛 Fixed page preview
issue #12444
2020-12-04 11:35:15 +01:00
Daniel Lockyer
a7e17c6b98
Changed URL protocol in oembed acceptance tests
- `https` was getting caught somewhere with nock and metascraper and
  caused each test case to hang for 3 seconds
- `http` still tests what we want and is instant
2020-12-03 21:46:01 +00:00
Kevin Ansfield
9ff7423b2b
Added email.open_rate order option to posts api (#12439)
refs https://github.com/TryGhost/Ghost/issues/12420

- updated `order` bookshelf plugin's `parseOrderOption()` method to return multiple order-related properties
  - `order` same as before, a key-value object of property-direction
  - `orderRaw` new property that is a raw SQL order string generated from `orderRawQuery()` method in models
  - `eagerLoad` new property that is an array of properties the `eagerLoad` plugin should use to join across
- updated `pagination.fetchAll()` to apply normal order + raw order if both are available and to handle eager loading / joins when `options.eagerLoad` is populated
- updated post model to include details for email relationship and to add `orderRawQuery()` that allows `email.open_rate` to be used as an order option
2020-12-03 20:13:37 +00:00
Daniel Lockyer
47397ca7a6
Updated logging message for webhooks
- removes double space and tidies up the output
2020-12-03 17:50:53 +00:00
Daniel Lockyer
04fb7f8e69
Refactored acceptance tests to use async-await
- refactoring the acceptance tests to use async-await removes all the Promise
  chaining we had, and streamlines the coding styles we have across the code so
  test files are more alike
2020-12-03 08:47:01 +00:00
Renovate Bot
30a98c0185 Update dependency grunt-mocha-cli to v7 2020-12-03 08:46:48 +00:00
Daniel Lockyer
4ef019d88d Deleted v3 regression tests
- at the time of writing, the v3 API === canary API
- we have both v3 + canary regression tests, which are nearly the same
  but there are slight deviations that we keep missing when adding new
  tests
- the canary tests are actually describing functionality of the v3 API
- therefore, we should be ok to delete the v3 regression tests for now
- when v3 is stable, we can copy the canary tests back to v3
2020-12-02 18:27:29 +00:00
Renovate Bot
2d621c9680
Update dependency sanitize-html to v2.2.0 2020-12-02 15:32:57 +00:00
Daniel Lockyer
07354ab1ec
v3.39.0 2020-12-02 15:11:33 +00:00
Daniel Lockyer
40f9bf3769
Updated Ghost-Admin to v3.39.0 2020-12-02 15:11:32 +00:00
Kevin Ansfield
a945ae7c02 Added sentry error capturing to email analytics jobs
no issue

- jobs operate in their own process and so do not share any of the error capturing that is configured in the parent process
2020-12-02 14:57:15 +00:00
Kevin Ansfield
c40a8e978d Fixed incorrect knex syntax
refs 166d072cd4

- accidentally committed an earlier attempt 🤦🏻‍♂️ updated with the valid syntax
2020-12-02 14:53:10 +00:00
Kevin Ansfield
166d072cd4 Fixed email analytics job not being registered when creating an email
no issue

- job registration was checking for submitted emails in it's email count but the job registration method is called as soon as an email is created meaning the email has a status of 'pending' which prevented the analytics job from being started until a second email was sent
2020-12-02 14:47:18 +00:00
Kevin Ansfield
f802128cfc
Added emailAnalytics config feature flag (#12443)
no issue

- email analytics may be desirable to fully switch off in certain circumstances, when that happens we want to prevent related background jobs from running and expose the feature flag via the config endpoint in the Admin API so that clients can adjust accordingly
2020-12-02 13:22:12 +00:00
Kevin Ansfield
2951eb9eaf Added missing moment require
refs 92ef83c61a
2020-12-02 12:15:27 +00:00
Kevin Ansfield
92ef83c61a Adjusted email jobs registration to check for emails newer than 30 days
no issue

- if emails are older than 30 days we wouldn't be able to fetch any analytics for them and if a site used emails in the past but is no longer using them it doesn't make sense to keep potentially expensive background worker threads spinning up
2020-12-02 12:14:22 +00:00
Daniel Lockyer
947603d0e3
Added original error to thrown error in mega code
- the original error is not propagated up with the error we throw, so it
  can sometimes be difficult to see what's going on down the line
2020-12-02 09:04:20 +00:00
Kevin Ansfield
d675278b0b
Prevented scheduling of recurring analytics jobs when not using emails (#12441)
no issue

- recurring jobs spin up worker threads which can be quite CPU intensive even when not performing much processing, this can be problematic in environments where there are many Ghost instances running
- updated the email job scheduling to be skipped on bootup when there are no emails in the database and to be started when the first email is created as long as we're not in testing env
- increase analytics job schedule from every 2 minutes to every 5 minutes to help spread the load further across instances
2020-12-02 08:17:44 +00:00
Renovate Bot
57c3244f3a Update dependency semver to v7.3.4 2020-12-02 00:07:11 +00:00
Kevin Ansfield
249fd4f06a Fixed email count check in email-analytics service
no issue

- raw knex `.count()` does not return a straight number, we need to handle an array of rowDataPacket objects
2020-12-01 15:02:04 +00:00
Renovate Bot
0e46898aaa
Update dependency @sentry/node to v5.28.0 2020-12-01 14:27:06 +00:00
Kevin Ansfield
641be8fec3 Adjusted email analytics job scheduling to avoid API spikes
no issue

- typically cron/later schedules will schedule for :00 on the minute which would create API spikes with every members-email-using Ghost site hitting the API at the same time
- adjusted the scheduling to use cron syntax with job runs every 2 minutes on 1,3,5... or 2,4,6... and a random seconds value to smooth usage across sites
2020-12-01 11:27:34 +00:00
Naz
ddcdaa6193 Bumped job-manager version to 0.3.4
no issue

- Fixes a bug with ignored seconds when specifying cron syntax
2020-12-02 00:24:03 +13:00
Rish
e02ef7f16d Pinned Portal to latest minor version
no refs

Ghost's Portal script is loaded via unpkg which was till now pinned to load `@latest` version, which unpkg auto-resolved to the latest released Portal version. This allowed fast iterations on Portal while still in active beta development to test latest Portal releases.

Going forward, Portal will be pinned to latest specific minor version that allows releasing new features that are not backward compatible without affecting older Ghost releases.

Note: All previous Ghost releases with Portal `@latest` will continue to resolve to latest version and will need to update to latest Ghost 3.x to use all Portal features.
2020-12-01 16:16:14 +05:30
Kevin Ansfield
db7fff0b2a Fixed misleading emailAnalytics.fetchLatest debug statement 2020-12-01 10:16:41 +00:00
Kevin Ansfield
d604f96a9c Added no-emails guard check when fetching latest email events
no issue

- replicates the same guard used in `fetchLatest()` to prevent contacting Mailgun when there are no emails in the database
2020-12-01 10:15:31 +00:00
Daniel Lockyer
622a57fc26
Added missing canary test present in v3 tests
- this test was present in the v3 test suite but not in the canary ones
- given v3 == canary currently, these tests should essentially be the
  same
- copied the test over
2020-12-01 09:35:24 +00:00
Kevin Ansfield
b35eef1b02 Added kill-switch config for email analytics background jobs
no issue

- it's possible background jobs may cause unintended side-effects so it's useful to have a kill-switch to disable them individually to keep sites working
2020-12-01 09:10:59 +00:00
Daniel Lockyer
8f86885132 Fixed typo in v2 oembed API
- canary uses `maxwidth` and it's the correct parameter to use
2020-12-01 08:59:41 +00:00
Kevin Ansfield
5c5bf2d3b0 Fixed oembed tests being network dependent
no issue

- some tests started failing locally because example.com was not resolvable
- mocked dns lookup for all tests so that we're always testing against expected public/private IP address blocks
2020-11-30 21:44:15 +00:00
Scott Cabot
da5586e5aa Fixed user import with no created_at date breaking graph
- users imported from CSV with no created_at date where having their created_at date being stored as an int rather than a datetime.
- this was causing parsing issues with the graph so this commit fixes the formatting
2020-11-30 11:21:22 +00:00