Commit Graph

11505 Commits

Author SHA1 Message Date
Nazar Gargol
f8adb4a672 Refactored bulk insert/delete operations into separate module
no issue

- Moved bulk db operations outside of importer module to create clearer separation of responsibilities
2020-08-20 18:42:38 +12:00
naz
3a594ce22e
Improved error handling for batch inserted member records (#12146)
no issue

- When batch insert fails handling should be more granular and aim to retry and insert as many records from the batch as possible.  
- Added retry logic for failed member's batch inserts. It's a sequential insert for each record in the batch. This implementation was chosen to keep it as simple as possible
- Added filtering of "toCreate" records when member fails to insert. We should not try inserting related members_labels/members_stripe_customers/members_stripe_customer_subscriptions records because they would definitely fail insertion without associated member record
2020-08-20 17:41:47 +12:00
Nazar Gargol
2e769e3122 Fixed handling for Stripe connected members import
no issue

- When stripe is disconnected and there are Stripe-connected records present in imported set they should not be processed and proper error should be thrown
2020-08-20 17:08:19 +12:00
Nazar Gargol
4498b4624a Fixed label serialization in members bulk importer
no issue

- After a refactor logic was missing trimming logic and handling for empty labels
2020-08-20 16:43:42 +12:00
Nazar Gargol
32fd913ac3 Fixed default subscribed value for member model in the importer
no issue

- The default value should not be a string but rather a boolean
2020-08-19 23:42:48 +12:00
Kevin Ansfield
d8e319af88 🐛 Fixed card spacing and caption styling for member emails in Outlook
closes https://github.com/TryGhost/Ghost/issues/12139

- once the email content has been rendered in the post serializer, perform some whole-content transformation of `figure` and `figcaption` to `div` using cheerio
- juiced will have already inlined the elements styles so there's no need to adjust the template's stylesheet
2020-08-19 12:34:14 +01:00
Nazar Gargol
616767d0ec Fixed members bulk import's createdBy field assignment
no issue

- Method signature has been changed with cada88975a, the call needed an update as well
2020-08-19 17:14:45 +12:00
Nazar Gargol
6512b98049 Updated flaky member import tests
no issue

- Because there's concurrency involved during member import the order cannot be guaranteed.
2020-08-19 16:43:58 +12:00
Renovate Bot
090d977c8e
Update dependency @tryghost/kg-markdown-html-renderer to v2.0.2 2020-08-18 18:05:37 +00:00
Renovate Bot
52c4292ad7
Update dependency @tryghost/kg-default-atoms to v2.0.2 2020-08-18 16:56:11 +00:00
Renovate Bot
daf79e367d
Update dependency @tryghost/kg-card-factory to v2.1.2 2020-08-18 15:36:49 +00:00
Daniel Lockyer
08a72d38a9 v3.30.0 2020-08-18 16:35:02 +01:00
Daniel Lockyer
78fe4f8bf9 Updated Ghost-Admin to v3.30.0 2020-08-18 16:35:02 +01:00
Jeremy Davidson
681c96b5e9
🐛 Fixed bookmark card author/publisher spacing in member emails (#12134)
closes #12078

- Root cause was that pseudo class .kg-bookmark-author:after was not getting inlined to email newsletter or its preview.
  - That is where the margin and bullet-point content are added between author and publisher.
- Dependency juice supports an option inlinePseudoElements which is false by default.
- Fix was to set inlinePseudoElements to true when serializing post email.

Co-authored-by: Jeremy Davidson <jeremy@crossingcontour.com>
2020-08-18 16:18:44 +01:00
Hannah Wolfe
9f291b2d10 Improved testmode logging of shutdown status
- testmode allows us to test Ghost's behaviour with long running requests and jobs
- moved all the testmode logging to only happen if Ghost starts successfully to make it clearer what is happening
- added a (currently very dirty) bit of code to fetch the jobService and output the queue status giving us a similar
     output for both open connections and jobs
- all of this allows for debugging exactly what Ghost is doing whilst it is processing a shutdown request
2020-08-18 15:48:42 +01:00
Kevin Ansfield
cefcdad090 🐛 Fixed emails appearing very wide in Outlook and improved email image sizes
closes https://github.com/TryGhost/Ghost/issues/11536

- bumps `@tryghost/kg-default-cards`
  - image and gallery cards now output `width/height` attributes on `img` elements with a max width of 600px
  - uses resized images where possible to keep email weight down
- adds `height: auto` style to image card images so that the `height` attribute does not cause distortion at smaller screen widths
2020-08-18 15:33:54 +01:00
Fabien 'egg' O'Carroll
cada88975a
Updated bulk importer to improve performance (#12128)
no-issue

* Added bulkAdd method to Member,Customer&Subscription model
  This allows us to keep the db access in the model layer
* Updated @tryghost/members-api to 0.27.2
  This includes fixes for rate-limiting of requests, and exposes necessary
  Stripe methods for creating customers and complimentary subscriptions,
  without affecting the database.
* Refactored importer to parallelise tasks where possible
  By parallelising our tasks we are able to improve the speed at which the
  entire import completes.
2020-08-18 13:39:45 +01:00
Daniel Lockyer
8fa9536122 Updated test labelling in GitHub Action workflow
- I'm expecting to add other v2 tests soon, so it's good to specify
  which v2 we're testing here
2020-08-18 13:10:34 +01:00
Daniel Lockyer
e8081f6691 Pinned Ubuntu version for tests & release workflows
- we were using `ubuntu-latest`, which is an alias for 18.04, but there's an
  increasingly likely chance that the default becomes 20.04 soon.
- we don't officially support 20.04 yet, so we want to pin to our
  supported version until we're ready
2020-08-18 13:08:22 +01:00
Daniel Lockyer
3a46aafcbb Merged 3.29.2 into master
v3.29.2

* tag '3.29.2':
  v3.29.2
  Updated Ghost-Admin to v3.29.2
  🐛 Fixed failing migration from <2.34 to 3.29
  🐛 Fixed published time and modified time for structured data (#12085)
  🐛 Fixed meta data when using tag data in collection (#12137)
  Updated Stripe JS to load async in head
  🐛 Removed [http://url/] output in member email preview text
  🐛 Fixed apostrophes not displaying correctly in Outlook for member emails
2020-08-18 12:17:35 +01:00
Daniel Lockyer
0b0e9aa5fc v3.29.2 2020-08-18 12:15:49 +01:00
Daniel Lockyer
0a65471008 Updated Ghost-Admin to v3.29.2 2020-08-18 12:15:49 +01:00
Kevin Ansfield
69612e38ff 🐛 Fixed failing migration from <2.34 to 3.29
closes https://github.com/TryGhost/Ghost/issues/12129

Migrations were failing when upgrading from a version that didn't include stripe member subscriptions table. The `up` migration was checking for the existence of a unique index that was added to the schema in 3.29 but it was using the wrong variable name which meant that it would never return true for an existing index. For most migrations this worked because the index existed but when through 2.34 the table was created from scratch and did included the index at that point.

- fixed variable name and re-ordered variable assignment for better code locality that would have made the typo more visible
2020-08-18 11:34:12 +01:00
Roshan Dash
c66304f66a 🐛 Fixed published time and modified time for structured data (#12085)
closes #12059

- Published Time and Modified Time were not populating for 'page' context because it is an extension of 'post' and hence there was no context 'page'. 
- Fixed it by using the common contextObject & `getContextObject` utility. 
- Should also fix some other missing parameters.
2020-08-18 11:34:06 +01:00
Fabien 'egg' O'Carroll
993e45bd88 🐛 Fixed meta data when using tag data in collection (#12137)
closes #12130 

When defining a collection with a tag as the data source, the metadata
was not correctly applied due to the context array not including 'tag'.

This update keeps the context management all in the same context helper
file and follows the same pattern as for posts/pages as a data source.
2020-08-18 11:34:00 +01:00
Rish
1a6eec40d1 Updated Stripe JS to load async in head
no issue

- Stripe JS is added to a theme via ghost_head if a Stripe account is connected to members enabled site
- Previously, the script was not loading async which blocked the main thread, changes the script load to async to avoid rendering block
- Members script is already being loaded with `defer` so does not block the main thread
2020-08-18 11:33:54 +01:00
Kevin Ansfield
f8f0f76788 🐛 Removed [http://url/] output in member email preview text
no issue

- we output the post excerpt in a hidden div in the email template so that email clients pick it up as the "preview" text when listing emails
- when no custom excerpt is provided the preview text is grabbed from post.excerpt which is the first 500 chars of the post.plaintext value
- post.plaintext formats links as "Link [http://url/]" which is unwanted in html email previews

- add a basic replacement to the post email serializer to remove any `[http://url/]` occurrences from the post excerpt before rendering the email content
2020-08-18 11:33:48 +01:00
Kevin Ansfield
f272b3fa5c 🐛 Fixed apostrophes not displaying correctly in Outlook for member emails
refs https://github.com/TryGhost/Ghost/issues/11536

- Outlook supports `&#39;` as a special char for apostrophes but not `&#apos;` which is what cheerio/juiced render
- adds a basic string placement to the email serializer to switch to the older style of special char
2020-08-18 11:33:43 +01:00
Renovate Bot
79180d5446
Update dependency @tryghost/members-api to v0.27.2 2020-08-18 10:31:44 +00:00
Rishabh Garg
2bac2c67cc
Granted Admin users ability to impersonate member (#12132)
refs #12126

- Adds migration to add impersonation permission to administrators
- Adds default permission fixture to allow administrators to read member impersonation urls
- Allows administrators to create member impersonation magic links
2020-08-18 14:17:56 +05:30
Nazar Gargol
6ca51eae29 Fixed member import csv validator for labels
no issue

- Labels can have a null value when there is none defined in the imported file. Such records fails for "string" type in JSON schema
2020-08-18 15:25:04 +12:00
Renovate Bot
c9cc19cb74
Update dependency @tryghost/members-api to v0.27.0 2020-08-17 20:04:08 +00:00
Renovate Bot
742f992601
Update dependency knex to v0.21.5 2020-08-17 18:16:56 +00:00
Kevin Ansfield
752c1022f9 🐛 Fixed failing migration from <2.34 to 3.29
closes https://github.com/TryGhost/Ghost/issues/12129

Migrations were failing when upgrading from a version that didn't include stripe member subscriptions table. The `up` migration was checking for the existence of a unique index that was added to the schema in 3.29 but it was using the wrong variable name which meant that it would never return true for an existing index. For most migrations this worked because the index existed but when through 2.34 the table was created from scratch and did included the index at that point.

- fixed variable name and re-ordered variable assignment for better code locality that would have made the typo more visible
2020-08-17 18:31:00 +01:00
Roshan Dash
c81d11b910
🐛 Fixed published time and modified time for structured data (#12085)
closes #12059

- Published Time and Modified Time were not populating for 'page' context because it is an extension of 'post' and hence there was no context 'page'. 
- Fixed it by using the common contextObject & `getContextObject` utility. 
- Should also fix some other missing parameters.
2020-08-17 15:52:31 +01:00
Fabien 'egg' O'Carroll
3430c47725
🐛 Fixed meta data when using tag data in collection (#12137)
closes #12130 

When defining a collection with a tag as the data source, the metadata
was not correctly applied due to the context array not including 'tag'.

This update keeps the context management all in the same context helper
file and follows the same pattern as for posts/pages as a data source.
2020-08-17 15:21:09 +01:00
naz
51c9a50c4f
Added validation layer to members import endpoint (#12131)
no issue

- Additional validation is needed for imported data because in case of bulk insertions (through knex) we bypass model layer validation - this could lead to invalid data in the database, which would be hard to fix.
- Chose validation method we use for other endpoints - through JSON Schema. It proved to be very performant (200ms overhead for 50k records). When comparing it with iterative method (validating each record separately) this was adding about 17s of overhead.
- Refactored returned values from "sanitizeInput" method to encapsulate more logic so that the caller doesn't have to calculate amount of invalid records and deal with error types
- Whole sanitizeInput method could now be easily extracted into separate module (somewhere close to members importer)
- Bumped members-csv package. It is meant to handle empty string values - '' and null, which should allow validating member records more consistently!
2020-08-17 23:28:57 +12:00
Renovate Bot
87e6954ef9
Update dependency @tryghost/members-csv to v0.3.0 2020-08-17 06:03:25 +00:00
Rish
c6cfadbfb7 Updated Stripe JS to load async in head
no issue

- Stripe JS is added to a theme via ghost_head if a Stripe account is connected to members enabled site
- Previously, the script was not loading async which blocked the main thread, changes the script load to async to avoid rendering block
- Members script is already being loaded with `defer` so does not block the main thread
2020-08-17 10:16:42 +05:30
Renovate Bot
c962afa5ab
Update dependency ajv to v6.12.4 2020-08-17 00:06:18 +00:00
Renovate Bot
b2c34029e1
Update dependency eslint to v7.7.0 2020-08-14 22:22:42 +00:00
Renovate Bot
78911257f4
Update dependency lodash to v4.17.20 2020-08-13 18:05:45 +00:00
Daniel Lockyer
046b5540fe Removed workaround for intermittent ghost-server logging
- upon further testing, this started to not work again so I'm removing
  this horrible hack
- it could be Bunyan playing around with us, and the server is actually
  shutting down
2020-08-13 18:45:51 +01:00
Renovate Bot
385af269a5
Update dependency @tryghost/url-utils to v0.6.20 2020-08-13 16:39:20 +00:00
Daniel Lockyer
d4f1274ba5 Returned empty promises in ghost-server
- I have no idea why this is required and I really don't like it
- this change looks superfluous but I had problems with Ghost not
  displaying shutting down messages properly and this is the only
  thing that got it working
- the async-awaits were getting stuck in `await this._cleanup()`,
  and then wouldn't enter into the `finally` block
- oddly, this was only reproducible when a job was running, it would
  shut down normally otherwise
- if I find the solution in the mean time, I'll happily get rid of this
2020-08-13 17:26:33 +01:00
Renovate Bot
35b9b7cc8c
Update dependency @tryghost/string to v0.1.11 2020-08-13 16:06:33 +00:00
Renovate Bot
1631fe5159
Update dependency @tryghost/social-urls to v0.1.12 2020-08-13 14:13:08 +00:00
Renovate Bot
38cd76f965
Update dependency @tryghost/html-to-mobiledoc to v0.7.3 2020-08-13 12:30:15 +00:00
Renovate Bot
2bb8d22de3
Update dependency @tryghost/helpers to v1.1.30 2020-08-13 11:05:41 +00:00
Nazar Gargol
ef41c57974 Added test coverage for invalid member import emails
no issue

- This test should serve as a control for new batched importer
2020-08-13 22:14:52 +12:00