refs https://github.com/TryGhost/Team/issues/598
We now have several pre-conditions related to members which determine
whether or not Ghost is allowed to start. Rather than burying this
within the members-api module, we have now surfaced them to an init
method which can be called during the boot sequence of Ghost. This will
allow us to exit early and explicitly.
no-issue
Our linter now requires that files named index.js have less than 50
lines, so this renames the index.js file to service.js and reexports
service.js from index.js so that linting will pass.
refs https://github.com/TryGhost/Team/issues/693
Since we've got rid of the concept of Complimentary with the Custom
Prices work, we're removing the 'comped' status from members. This
involves a migration for existing members, a schema update for the
validation, and a bump to members-api to no longer use the 'comped'
status for new members.
We also update the aggregation of the MemberStatusEvent to consider the
'comped' status as 'paid', and that there are 0 'comped' status events
in the database.
We can consider a migration for this data in the future, either adding
new status events moving from 'comped' to 'paid', or by modifying
existing status events. However both of these are very difficulty to
write a down migration for, and might be best saved for a major version.
- @tryghost/members-api@1.7.0 is the version that includes the required
changes, however we have already bumped to 1.8.0 in Ghost
refs https://github.com/TryGhost/Team/issues/693
Since we no longer have the concept the "comped" we update the v3 API to
always have a `comped` flag of `false` - maintaining backwards
compatibility.
refs a4c78dbf19
Updates member data on edit to include products data when comped status is changed, as by default we don't include products data when member goes from free to paid subscription due to comped being added.
closes https://github.com/TryGhost/Team/issues/699
With custom products, saving a member with subscriptions on member detail page in Admin throws errors on console, though the save is successful. This breaks the Admin as user needs to refresh the screen again to get rid of error. This change -
- updates the response on member save to return `price` object in subscription
- updates tests
no issue
Keeping CSRF enabled there would prevent oauth from working as users are redirected from the provider domain to the /callback route, where they are logged-in
no refs
Filters active prices in Portal settings to only contain the selected prices by site owner in new monthly/yearly price id settings, ignoring all other prices for now.
no refs
Since backend now allows multiple prices but we want the prices to be currently limited to monthly/yearly on UI, we need new settings to store the current monthly/yearly price by the site owner. These settings determine the active prices shown in Admin / Portal for the site till we allow all custom products/prices again.
closes https://github.com/TryGhost/Team/issues/675
Outlook will display images at their native resolution if no `width` attribute is supplied. Content images were fixed a while ago but feature images would still render very wide and cause horizontal scroll and text size/alignment issues.
- modify `post.feature_image` and add a `post.feature_image_width` property before passing it through to the email template
- for Unsplash images we assume all images are larger than 600px so we change the URL to reference a 1200px image and set the image width to 600 (to keep images on retina displays crisp)
- for other images we probe the image to fetch the original dimensions and give set an image width of 600 if needed, if it's a locally-hosted image we update the URL to point at a max 1200px version
- updated email template to output a `width` attribute on the feature image `<img>` tag if it's set
no issue
- `getLocalSize()` is useful outside of the mobiledoc populate-image-sizes function
- expanded `ImageSize` class with new methods
- `getOriginalImageSizeFromStoragePath()` - takes the "original" image extraction and test from `getLocalSize()` and makes it more generally available
- `getImageSizeFromStorageUrl()` - takes the path extraction from `getLocalSize()` to make image sizes from local urls more generally available
- `getOriginalImageSizeFromStorageUrl()` - URL version of the new `getOriginalImageSizeFromStoragePath()` method
closes https://github.com/TryGhost/Team/issues/560
closes https://github.com/TryGhost/Ghost/issues/12870
The endpoint `/members/api/member/` is used by Portal for fetching member details on site load to setup different flows. The response from this endpoint for logged out member has now changed from 401 Unauthorized to 204 No Content.
Ghost API was previously returning 401 Unauthorized error for logged-out member as this seemed to be technically correct response for unauthorized access to membership features. This resulted in a lot of confusion for end users where visible 401 errors on console were perceived as errors in the script as well as caught by loggers as erroneous traffic. Also for an end user, in the context of visiting a website - the user themselves is not trying to gain access to anything so this becomes cause for more confusion.
After internal discussion, the endpoint - [SITE_URL]/members/api/member- now returns 204 No Content instead of 401 for logged out member, denoting server was able to process the request but did not find any associated member. This should avoid any unwanted error logging on Portal load on a site, as well as make Portal functioning more transparent for a site.
no-issue
When importing Members it is possible to have both the
complimentary_plan and the stripe_customer_id columns set, this can
result in unusual outcomes, for example when importing a customer with a
zero-amount subscription, they would end up with two "comped"
subscriptions, and there would be two "comped" prices in the database.
As we are deprecating the use of "comped" in favour of creating a
subscription with a specific price, we're updating the import to prefer
`stripe_customer_id` column, only using the `complimentary_plan` column
when it is the only of the two columns passed.
refs https://github.com/TryGhost/Team/issues/581
closes https://github.com/TryGhost/Team/issues/582
Emails can now be sent to members with specific associated labels or products by specifying an NQL string. We want to bring the same members segment feature to content by allowing `visibility` to be an NQL filter string on top of the `public/members/paid` special-case strings.
As an example it's possible to set `posts.visibility` to `label:vip` to make a post available only to those members with the `vip` label.
- removed enum validations for `visibility` so it now accepts any string or `null`
- bumped `@tryghost/admin-api-schema` for API-level validation changes
- added nql validation to API input validators by running the visibility query against the members model
- added transform of NQL to special-case visibility values when saving post model
- ensures there's a single way of representing "members" and "paid" where NQL gives multiple ways of representing the same segment
- useful for keeping theme-level checks such as `{{#has visibility="paid"}}` working as expected
- updated content-gating to parse nql from post's visibility and use it to query the currently logged in member to see if there's a match
- bumped @tryghost/members-api to include label and product data when loading member
refs https://github.com/TryGhost/Team/issues/667
On clean and existing installs, the default product created should be named the same as the site title in the first setup so the UX on Portal and everywhere is consistent. This change adds a migration to update existing sites which already have a default product created via fixture, and rename them to their current site title. The rename is only done if the Product name is still the same as in fixture - `Default Product`.
refs https://github.com/TryGhost/Team/issues/671
When turning on custom products, existing sites should have default price descriptions that match existing values for prices. This change sets the default description for Free price to match existing hardcoded value.
refs https://github.com/TryGhost/Team/issues/667
On clean and existing installs, the default product created should be named the same as the site title instead of the name in fixture. This change updates the default product's name to site title during the site setup. We use the Product name in Portal.
refs https://github.com/TryGhost/Team/issues/637
With custom products it's possible to change the name and description of any price. This assumes that people would want to change the same properties of a Free membership, and wires up the values for free membership price settings to Portal site settings API for Portal UI
no-issue
Our base model will only automatically convert numbers to booleans if
the type is 'bool' - however this column was incorrectly added with a
type of 'boolean'. Lucklily - knex with both MySQL & SQLite3 will add
a column with the same type for both of these, so no migration is needed
to fix it.
refs https://github.com/TryGhost/Team/issues/637
The "free" price - when Members signup without using Stripe, should have
a name and description, so that it can be displayed in Portal in a
similar way to paid price's. As there is only ever one, and it is not a
fully fledged price, a setting makes more sense than a dedicated db
table.
refs https://github.com/TryGhost/Team/issues/586
We are no longer using the `stripe_plans` setting, instead we are using
the `stripe_prices` database table. However, we must keep the setting as
the migration from the setting to the database is not done as a standard
migration, but in code. This means our code has to still read and pass
the setting because we will never know if the migration in code has run
yet.
The `portal_plans` setting has been updated to only include 'free' by
default, because the setting must include id's now rather than names.
refs https://github.com/TryGhost/Team/issues/588
refs d72ba77aba
- When limit is in place we don't want to allow sending out a new batch of emails if it would go over limit
- See referenced commit for example configuration
refs https://github.com/TryGhost/Team/issues/588
- This is a new type of limit allowing to measure resource use (e.g. sent emails) per period (e.g. subscription, billing, cycle, etc)
- To enable periodical limit add following values under `hostSettings.limits`:
```
"emails": {
"maxPeriodic": 10,
"error": "Your plan supports up to {{max}} emails. Please upgrade to reenable sending emails."
}
```
and following under `hostSettings.subscription`:
```
"subscription": {
"start": "2020-04-02T15:53:55.000Z",
"interval": "month"
}
```
- Above config would allow checking if 10 emails per month starting on the 2nd of every month has been reached untill now
refs https://github.com/TryGhost/Team/issues/665
Portal only needs to work with active prices(not archived), this change filters prices sent to Portal to only include active prices
refs https://github.com/TryGhost/Team/issues/586
A discussion in the Members team resulted in us determining that we do
not need to enforce unique names for Products. Stripe does not enforce
uniqueness for their Products, and we feel it's not necessary for us to.
refs 37ebe723c6
- `package-json` was a standalone library using dependency injection so
we could pull it out into its own package in Utils
- this was done in the commit referenced above
- this commit removes the implementation and tests in Ghost and replaces
the require in the initialization wrapper with the new package
refs https://github.com/TryGhost/Team/issues/581
refs https://github.com/TryGhost/Team/issues/582
When publishing a post via the API it was possible to send it using `?email_recipient_filter=all/free/paid` which allowed you to send to members only based on their payment status which is quite limiting for some sites.
This PR updates the `?email_recipient_filter` query param to support Ghost's `?filter` param syntax which enables more specific recipient lists, eg:
`?email_recipient_filter=status:free` = free members only
`?email_recipient_filter=status:paid` = paid members only
`?email_recipient_filter=label:vip` = members that have the `vip` label attached
`?email_recipient_filter=status:paid,label:vip` = paid members and members that have the `vip` label attached
The older `free/paid` values are still supported by the API for backwards compatibility.
- updates `Post` and `Email` models to transform legacy `free` and `paid` values to their NQL equivalents on read/write
- lets us not worry about supporting legacy values elsewhere in the code
- cleanup migration to transform all rows slated for 5.0
- removes schema and API `isIn` validations for recipient filters so allow free-form filters
- updates posts API input serializers to transform `free` and `paid` values in the `?email_recipient_filter` param to their NQL equivalents for backwards compatibility
- updates Post API controllers `edit` methods to run a query using the supplied filter to verify that it's valid
- updates `mega` service to use the filter directly when selecting recipients
refs https://github.com/TryGhost/Team/issues/581
Editors are allowed to restrict post visibility and send emails to particular member segments, they need to be able to read labels so that they can select them in a member segment.
refs https://github.com/TryGhost/Team/issues/637
refs 75169b705b
With custom prices, Portal now needs to show all available custom prices in the UI as well as product's name and description in the Portal UI. This change adds product information to member site settings for Portal UI.
refs https://github.com/TryGhost/Team/issues/586
refs aa12770329
Using `id` as ghost id for subscription prices can be confusing as everything in the method refers ids to be stripe ids. This change updates the ghost id value to use `price_id` key in the serialization
refs https://github.com/TryGhost/Team/issues/496
We want to give more control over the default selection of email recipients when publishing a post, to do that we need somewhere to store those settings. These settings are site-wide and intended for use by admins to control the default editor behaviour for all staff users. They _do not_ control API behaviour, if you want to send email when publishing via the API it's still necessary to explicitly opt in to that using the `?email_recipients_filter=` query param.
- new `editor` settings group to indicate that these settings only affect the UI rather than the API
- `editor_default_email_recipients` controls overall behaviour, string/enum with these allowed values:
- `'disabled'`: no option to send email is shown in the editor's publishing dropdown
- `'visibility'`: (default) selected member segment is dynamic and matches the post visibility filter
- `'filter'`: specific member filter defined in `editor_default_email_recipients_filter` setting
- `editor_default_email_recipients_filter` is an NQL string for selecting members, used when `editor_default_email_recipients` is set to `'filter'`
- default value is `'all'`
- the segment string can be any valid NQL filter with the additional special-case values of `'all'` and `'none'`
no issue
- `Error` is very generic for this case and `IncorrectUsageError`
will populate the resulting error with the correct error code
- the `message` was pulled out to its own statement so we can avoid long
lines
no issue
- we're preparing the `package-json` lib to be extracted out of Ghost into
its own package so moving the initialization wrapper outside of the
folder makes the process a lot easier
refs https://github.com/TryGhost/Team/issues/586
refs 33f26fbf32
As part of serializing subscriptions with prices, we previously attached only the stripe price id to the price object for subscription. This change updates the price object to include both Ghost id and stripe price id for the object, as Portal needs to check the Ghost price id for logged in members to verify their current plan.
- calling i18n as a global const like this requires it to be loaded before anything else, when we have to manage this with the init() flow
- wrapping it inside the function where it's used ensures we don't call i18n til we need it
- also improved the i18n called without init error to include the key it was called with
- We only require a single value from urlUtils, the url for the site
- Move that logic back to the boot file makes it much more explict
- Will help if we want to refactor how urlUtils works, or when we want to move ghost-server out of core
- single authors were deprecated in v1.22 when we added multiple authors
- we always thought we'd clean this up a lot sooner, but it's stuck because it's an annoying thing to break people's shit over
- still saying "remove in vX" isn't useful, we need to know how long a feature has been deprecated so we can judge whether it's safe to remove
refs https://github.com/TryGhost/Team/issues/637
With custom prices, Portal now needs to show all available custom prices in the UI instead of just `monthly` and `yearly` prices. This change adds a list of all custom prices to Portal site settings for the default product which Portal will use to show the available prices in UI.
Note: As part of cleanup, the stripe price ids will be removed from the prices list.
Also:
- Fixes product name in serialised subscriptions
- Adds `type` value in serialised price object
refs https://github.com/TryGhost/Team/issues/586
The `products` and `stripe_prices` tables are missing a description
column which will be used by Portal to display information about the
products and prices
- Preparing to cleanup / change how we use events across Ghost
- Removing this unused bit of additional complexity makes it easier to reason about what we need
refs 829e8ed010
- i18n is used everywhere but only requires shared or external packages, therefore it's a good candidate for living in shared
- this reduces invalid requires across frontend and server, and lets us use it everywhere until we come up with a better option
refs https://github.com/TryGhost/Team/issues/579
The previous `members_allow_free_signup` -> `members_signup_access` migration made a direct correlation between the toggle `true/false` to `all/invite` under the assumption that behaviour between the two settings would be identical. The assumption was incorrect and the behaviour is changing so `invite` forces invite-only mode, stopping all front-end signup to free or paid plans with the free plan now being disabled via the portal plans setting.
- check existing `members_signup_access` setting and if it's `'invite'` migrate it to `'all'` where signup should still be possible. The "invite-only" mode should only be active if certain conditions are met:
- Stripe is not configured ("allow free member signup" off and no Stripe showed "invite-only" in portal)
- Stripe is configured but no plans are selected in portal (no plans showed "invite-only" in portal)
- when migrating `'invite'` to `'all'`, also remove `'free'` plan from the `portal_plans` setting to avoid previously paid-only sites unexpectedly showing a free plan on signup
refs https://github.com/TryGhost/Team/issues/634
- find earliest backup file created when a 4.3 migration was run, if found use the `members_allow_free_signup` value from there to change `members_signup_access` from `'all'` to `'invite'` if necessary
- Having these as destructured from the same package is hindering refactoring now
- Events should really only ever be used server-side
- i18n should be a shared module for now so it can be used everywhere until we figure out something better
- Having them seperate also allows us to lint them properly
refs https://github.com/TryGhost/Team/issues/588
- This check allows for a on/off switch to be set up on the instance and control limits around sending emails
- An example configuration for such check would look like following in config's hostSettings section, e.g.:
```
"emails": {
"disabled": true,
"error": "Email sending has been temporarily disabled whilst your account is under review."
}
```
refs https://github.com/TryGhost/Team/issues/637
refs https://github.com/TryGhost/Team/issues/591
We need to run migrations which will update the `portal_plans` setting
to use id's rather than names. This migration relies on the
`stripe_prices` table being complete populated. The migration to
populate the `stripe_prices` table was not added as a "normal"
migration because it needs to access the Stripe API over the network.
Any migrations that rely on this are unable to be run in a "normal"
migration as that cannot be sure that the database is in the correct
state.
The `portal_plans` setting migration is therefore run in code, and needs
access to the Settings model in order to modify the database.
closes https://github.com/TryGhost/Team/issues/645
- we did some refactoring in [1] to turn promise chained code into
async/await, but this removed an early `return` from the code
- therefore we'd continue on to further code, which breaks for obscure
reasons that weren't apparent from the error
- this commit adds back a return at the end of the block where we handle
staff API tokens to match the same functionality as before
- this is regression that landed in 4.3.0 and would break staff user
tokens
[1]: b677927322 (diff-bc0bedcac8ec9646d0644c86a91e46f4759bc1b0c2aebac54a2b26ec474c3d15L148-L155)
refs https://github.com/TryGhost/Team/issues/634
- the migration moving `members_allow_free_signup` to `members_signup_access` was expecting a raw boolean setting value but the actual value is a string so always evaluated as truthy making all sites look like they had "allow free members signup" toggled on when generating the new setting's value
- updated to check for an explicit string value in `up` and set an explicit string value in `down`
refs https://github.com/TryGhost/Team/issues/579
The new signup access setting allows site owner to set the type of access level allowed for a member which Portal needs to handle
refs https://github.com/TryGhost/Team/issues/579
`members_signup_access = 'invite'` now forces invite-only mode so both free and paid setups both use the `'all'` setting. To ensure we're properly allowing/disabling free (self signup) signups in the members API we need to update `allowSelfSignup()` to take additional settings into account.
- `true` when Stripe is not connected. There are no paid plans available in this configuration so free signup is always enabled. To disable free signup on a site with no Stripe setup the members signup access should be set to `invite` or `none`.
- `true` when Stripe is configured and free plan is enabled in portal, without it Members API would not send magic link emails to signup requests
- `false` in all other situations such as invite-only and members-disabled signup access modes, or when the free plan has been disabled in portal configuration
refs https://github.com/TryGhost/Team/issues/634
- the migration moving `members_allow_free_signup` to `members_signup_access` was expecting a raw boolean setting value but the actual value is a string so always evaluated as truthy making all sites look like they had "allow free members signup" toggled on when generating the new setting's value
- updated to check for an explicit string value in `up` and set an explicit string value in `down`
refs: bf0823c9a2
refs: ae86254972
- continuing the work of splitting up the theme service into logical components
Themes Service
- The serverside theme service now serves just the API and boot
- It loads the theme and passes it to the theme-engine via the bridge
This achieves the bare minimum goal of removing all the cross requires between server and frontend around themes
There is still a lot more to do to achieve an ideal architecture here as laid out in ae86254972
refs https://github.com/TryGhost/Team/issues/616
We need a way to assign Products to Members via a Subscription, and we've
followed the same pattern as the editSubscription method for the Members API
controller, which acts upon Subscriptions as a nested resource.
Subscriptions now are linked to products, and we've included those links by
default in the Member Admin API as we already include subscriptions by
default, and Products are now a core part of the Members feature-set.
- Modules in /shared are supposed to be standalone modules that can be required by the server or frontend
- As the server shouldn't require the frontend, and vice versa, shared modules should require neither
- Otherwise it just becomes a crutch for allowing cross-depenencies, and will create circular dependencies
The Bridge
- The bridge file is not meant to be a crutch sat allowing cross-dependencies, but rather a new component that manages the flow of data
- That data flows from the server/boot process TO the frontend, and should not flow in the other direction
- The management of that flow of data is necessarily hacky at the moment, but over time the architecture here should get clearer and better
- Still, for the time being it will need to handle requiring across components until that architecture matures
- Therefore, it should live in core root, not in core/shared
refs: bf0823c9a2
- Added a new bridge class that lives in shared. This should eventually be responsible for all cross-communication between the frontend and the server
- Having all the gnarly shared bits in one place should help us refactor more easily
- For now it also reduces requires between the core/server and core/frontend folders that are meant to be separate
- All calls to getApiVersion have also been renamed to getFrontendApiVersion, as this is different to the "default" API version
- Slowly getting to the point where frontend/services/themes can be moved to server/services/themes :)
refs: bf0823c9a2
- continuing the work of splitting up the theme service into logical components
- this is where it starts to get fiddly as the getActive function in themeService index is required across the frontend/backend mostly due to its use in the getApiVersion method
- for now left one usage of the getActive method in place in ghost-locals middleware ready for the next phase of the refactor, which will move some of the themeService index into a shared location
issue https://github.com/TryGhost/Team/issues/614
- Users who have a password can directly sign-in via oauth
- User who are logged-in get their password disabled
- Users accepting an invitation get their password disabled
- The way we disable password is by setting it to a long random password
refs c873899e49
- as of `bson-objectid` v2.0.0, this library exports the function
to generate an ObjectID directly, and then you need to use `.toHexString()`
to get the 24 character hex string - 6696f27d82
- this commit removes all uses of `.generate()` and replaces with this
change
refs https://github.com/TryGhost/Team/issues/619
As part of the Custom Products work, we are linking members to products
when updating their subscriptions. This requires that we have at least
one product in the database. For existing sites that are using Members
this is handled by the v4.3 03 migration. But for new sites we must
include a fixture.
Also fixes the tests to not reply on the order of the fixtures
refs https://github.com/TryGhost/Team/issues/616
This is a basic scaffold of the API to get things moving.
The input serializer is so that the controller logic does not need to
know about the json-api shape of the input data.
The output serializer is an adaptation of the members one.
refs https://github.com/TryGhost/Team/issues/586
- Adds new `stripe_price_id` column to subscriptions table to store stripe price ids with `index`
- Populates `stripe_price_id` column value to current `plan_id` making the `plan_*` values redundant
- Updates tests
refs https://github.com/TryGhost/Team/issues/586
If a product inside Ghost is deleted, we want to cascade delete all associated Stripe products and prices as they always need to refer back to a ghost product and will hang without any reason otherwise. This change adds cascade delete for products -> stripe_products -> stripe_prices to avoid broken states
- This is the beginning of splitting up the theme service into:
- Storage components used by the API (should be a server service)
- Theme engine & rendering components used by the frontend (this new engine service)
- The code to activate a theme which is shared code where the API & frontend need to communicate
- This is needed because currently the frontend theme service is required and used by the API, creating tight coupling.
- In my quest to truly separate the API and frontend, this is one of many battles that needs winning
refs https://github.com/TryGhost/Team/issues/579
- setting `members_signup_access` to `'none'` effectively disables all built-in members functions on the front-end so setting `@labs.members` to `false` allows themes to react accordingly
- `@labs.members` keeps backwards compatibility with pre-4.0 versions where themes were using it to toggle member-related functionality
refs https://github.com/TryGhost/Team/issues/579
Currently the members signup setting is explicitly yes/no to allowing free members signup, with the implication that when set to "no" members is still active but members have to be created via Stripe or the admin API.
This change renames the setting and changes its type to allow more than a binary option.
- migration to create/update the new setting based on the old value
- free signup = "all", no free signup = "invite"; matches the current UI for this setting
- rename setting everywhere it's used/tested against
- modify `getAllowSelfSignup()` used to configure members packages to only return `true` when the new setting is set to `'all'` to match behaviour to the older setting
- update importer to rename the setting when importing from an older Ghost version
refs https://github.com/TryGhost/Team/issues/618
- The `oauth_client_id` and `oauth_client_secret` are placeholders to store OAuths related data.
- The flag for `oauth_enabled` or anything along those lines was not added intentionally in favour of checking if the `oauth_client_id` & `oauth_client_secret` are null.
refs https://github.com/TryGhost/Team/issues/616
All roles which can publish posts should be able to read/browse products, as content gating
will be based on products going forward.
Creating, updating & destroying products will often make modifications to Stripe which requires
Administrator or Owner roles.
We also improve the permissions tests so that we no longer rely on things being in a particular
order.
refs https://github.com/TryGhost/Team/issues/586
We have to use `belongsToMany` because of the way bookshelf relations
work. In reality the relationship is 'hasMany', e.g. a Product has many
Stripe Prices.
These relations are the minimal needed to satisfy the following
relationships without transforming the results. (e.g. flattening the
StripePrices from a list of StripeProducts for a Product)
Product -> StripeProduct: product.related('stripeProducts')
StripeProduct -> StripePrice: stripeProduct.related('stripePrices');
Product -> StripePrice: product.related('stripePrices');
StripePrice -> Product: stripePrice.related('stripeProduct.product');
refs https://github.com/TryGhost/Team/issues/586
- Passes new Product, Stripe Price and Stripe Product models to members API service
- Allows members service to populate the tables for existing plans and products
refs https://github.com/TryGhost/Team/issues/586
- Change the interval column to be `nullable` as one time payments won't have any interval
- Remove the `livemode` column as we store the connected account's livemode status at top level
https://github.com/TryGhost/Team/issues/599
- When custom integration limit is enabled all webhooks belonging to integrations have to be disabled as well. The result is the webhook would stop working and to discover that a user would need to navigate to Admin UI (this changes is yet to come, see refed issue)
https://github.com/TryGhost/Team/issues/599
- When custom integration limit is enabled all requests from existing integrations should not be accepted. With the exception of internal integrations like backup and scheduler
refs https://github.com/TryGhost/Team/issues/599
- This is a precursor change to tests which verify the hostSettings limits are working correctly
- Bumped limits-service version which allows for multiple calls of loadLimits on the same service instance
no issue
- There is a valid subset of statuses that can be set for the users but there's no "isIn" validation for possible values
- Additionally some of the statuses like warn-1, warn-2, etc. don't have a clear usecase (or at least nothing was found in the codebase for them to be used). They might be up for removal if this assumption is correct
refs https://github.com/TryGhost/Team/issues/586
- Member model now has `products` relation, sorted using `sort_order`, following convention from `labels`
- Product model has handling to set `slug` from name, following convention of Label model
- Updated filter plugin to handle filtering Member models by their `product` relations e.g. `product:[slug, slug]`
refs https://github.com/TryGhost/Team/issues/586
- Add the `stripe_products` table, so that we can map Stripe Products to Products in Ghost
- Add the `stripe_prices` table, so that we can associate Stripe Prices to Products table
refs https://github.com/TryGhost/Team/issues/586
- Add the products table, so that we can store Products in Ghost
- Add the members_products table, so that we can associate Members w/ Products
- Use sort_order on the members_products table to follow the same convention in members_labels
- Populate the products table with a single product, using the name from the stripe_product_name setting
- Populate the members_products table with relations based on the status column of the members table
Populating the tables allows us to transition from the current system, which does not care about products, into the
new system, where Products are used to group members. The intention is that all existing paid members have the
same product
refs https://github.com/TryGhost/Team/issues/585
- adds `DELETE /members/` route to the Admin API
- supports `?filter`, and `?search` query params to limit the members that are deleted
- `?all=true` is required if no other filter or query is provided
- uses `models.Member.bulkDestroy` which _will not_ cancel any Stripe subscriptions if members have them but _will_ clean up the Stripe relationship data in Ghost's database
refs https://github.com/TryGhost/Team/issues/510
- This version bump includes follwing interface improvements of the limit-service package: passing in errors as a parameter to "loadLimits" and allowing for custom "currentCountQuery" method implementations per limit
closes https://github.com/TryGhost/Team/issues/595
Due to a bug in `mrr_delta` calculation, we ended up reducing the MRR delta by twice the original amount when a subscription goes from active to canceled and storing it in `members_paid_subscription_events` table, which is used to show the MRR chart on Dashboard. The way we identify the incorrect events in the table which got the double negative value is by checking if they match certain criteria - Both `from_plan` and `to_plan` have same value as a subscription changes status while being on same plan.
This migration halves the `mrr_delta` for incorrect events to restore the correct MRR change for the site.
no refs
- Removes old `/members/stats` endpoint in favor of new `/members/stats/count` in canary/v4 which captures members counts using new events table
- Removes tests for old `/members/stats` endpoint
- Added test for new `/members/stats/count` endpoint
refs https://github.com/TryGhost/Team/issues/555
- Export files included a lot of data which was not used in the importer, for example: members, labels, migrations and many more. This lead to a lot of clutter in the import files and made it hard to reason about their purpose.
- The main purpose of exports - is to export importable resources. These are posts, tags, and users. The rest of data like members or migrations either have their own importer (like CSV importer for members) or does not and should not have any ways to be imported.
- These changes are in now way complete. It's a first step towards resource-based exports which could be properly versioned in the future on API level and not be a mirror of the DB structure.
- This is sort of a breaking change. But we are doing it because: (1) its an internal API that should not be used by external clients, (2) there was no public contract to have this API stable at any point, (3) we really need to get back the control over export files structure and size
- In case an external client was dependent on some structure of the exported json file they can still pass in ALL of previously exported data by passing table names in `include` query parameter.
refs https://github.com/TryGhost/Team/issues/555
- Previous blocklist approach was resulting in adding every single new table into an export automatically. Which creates possibility to leak sensitive data if not used porperly. Allowlist approach gives better control over what is exported, makes this information explicit, and version-control friendlier
refs https://github.com/TryGhost/Team/issues/555
- The getVersionAndTables was doing too much and was only used once creating clutter in doExport method. Refactored code doing 2 direct calls instead of destructuring "dances".
refs https://github.com/TryGhost/Team/issues/397
The default newsletter/support email address for a site is currently setup as noreply@DOMAIN , which means for a custom domain setup with www the email address becomes noreply@www.somesite.com which is not the expected behavior normally. This removes the `www` subdomain if present for those email addresses, but doesn't change any other subdomain
refs https://github.com/TryGhost/Team/issues/536
From 4.0, we ensure and require that accent colour is always set. This change removes hardcoded accent color fallbacks to avoid confusion as well as cause accidental fallback that is undesired causing themes to look different
closes https://github.com/TryGhost/Team/issues/571
- `post.plaintext` values were being transformed with the markdown URL transformer but that wasn't picking up the link format used in our plaintext fields resulting in absolute URLs being stored in the database rather than `__GHOST_URL__` URLs
- meant that if the `url` config is changed then plaintext and other calculated fields that used it would have URLs that referred to the old domain rather than the new one
- re-saving the posts would have updated it but that's not feasible to do manually for large sites
- bumped `@tryghost/url-utils` to a version that has plaintext transform utils and updated the post model's transform map
refs https://github.com/TryGhost/Team/issues/510
- contributors don't count towards the staff limit, therefore they should be allowed to be unsuspended
- currently, we don't check the role when unsuspending, which is incorrect
- this bug is pure oversight!
closes https://github.com/TryGhost/Ghost/issues/12791
closes https://github.com/TryGhost/Team/issues/566https://github.com/TryGhost/Ghost/pull/12787 introduced a significant performance regression due to a misunderstanding of when Bookshelf calls `.format()` ([related upstream issue](https://github.com/bookshelf/bookshelf/issues/668)). We expected `.format()` to only be called on save but it's also called when Bookshelf performs fetching and eager loading which happens frequently. `.format()` can be a heavy method as it needs to parse and serialize html and markdown so it should be performed as infrequently as possible.
- override `sync()` in the base model so we can call our own `.formatOnWrite()` method to transform attributes on `update` and `insert` operations
- this was the only feasible location in Bookshelf I could find that is low enough level to not require modifying model instance attributes
- gives models the option to perform heavy transform operations only when writing to the database compared to the usual `.format()` method that is also called on fetch in many situations
closes https://github.com/TryGhost/Team/issues/570
When site owner/admin updates their newsletter/support email address from settings, they receive an email with confirmation link which on success takes them to Ghost Admin on email settings screen with a toast about success. Since the path for email settings in Ghost Admin changed in v4, the fix updates the redirect link to new Admin settings URL.
refs 4dc413d6a1
- Fixed failing test cases which were designed to check non-major upgrade messages. There's no clear use cases for those, but still worth keeping such case in mind
closes https://github.com/TryGhost/Team/issues/564
refs https://github.com/TryGhost/Ghost/issues/10236
- The notification to upgrade to new 4.0 Ghost version was still visible to users after upgrading the instance to 4.0. This was caused by notification filtering not taking into account 3.x or 4.x versions.
- The fix filters out notifications that detect a major version notification using `x.0 is now available` pattern and compares current version to that major. This should future proof the issue from happening in Ghost 5.0 (but a proper holistic fix is preferable!)
closes https://github.com/TryGhost/Team/issues/552
Refactors URL transforms so they take place at the model layer rather than the API serializer layer. Continuation of the pattern created for the settings model in https://github.com/TryGhost/Ghost/pull/12738
- Added checks to all front-end tests to ensure output does not contain the magic replacement string
- includes failing acceptance test for `__GHOST_URL__` appearing in sitemaps
- Removed all transform-ready URL transforms from API serializers
- input serializers transform image urls relative->absolute to keep absolute-urls as the consistent "outside of the database" format
- output serializers should not need to perform any URL transforms as that will be done at the model layer
- Added url transforms to models layer
- removes knowledge from the API serializers which shouldn't need to know how data is stored internally in the database
- makes absolute urls the consistent "outside of the database" URL format
- adds transform step to the sitemap generator because the data used for that is fetched directly via knex which will not run through the bookshelf `parse()` methods
closes https://github.com/TryGhost/Team/issues/557
After updating to v4 on SQLite from v3, we were unable to delete members due to a foreign key mismatch error. This is because the migrations which recreates the tables for `members_stripe_customers_subscriptions` and `members_stripe_customers` doesn't add back the unique constraint which is needed by FKs for reference.
The migration creates the missing UNIQUE constraints on the tables.
refs 6b07d4b2a0
- The model is needed here, because it contains full set of fields. In some cases, like email-preview, the "plaintext" field is not present in "attrs" which causes the logic to fail.
- This should be sorted along with https://github.com/TryGhost/Ghost/issues/10396
refs https://github.com/TryGhost/Team/issues/467
refs a6f5eb71be
- When a generated excerpt is calculated for posts/page resources it uses raw model! to get the data. Model contains untranformed __GHOST_URL__ markup which has to be additionally processed before extracint an excerpt or use the transformed `plaintext` from available attributes (chose the latter to decrease complexity)
- Removed model dependency as `attrs` at this point of serialization should always contain the `plaintext` field. It's ugly and has an unsolved bug report here - https://github.com/TryGhost/Ghost/issues/10396. The reliance should be solved at some point, but definitely not a part of this issue
refs: https://github.com/TryGhost/Team/issues/510
refs: e30b9735fa
- In the case that there is a staff user limit, and the limit is maxed out, it is no longer possible to invite new staff users
- However, Contributors are not considered staff users and therefore it should always be possible to invite new Contributors
no issue
Comped members were not able to view paid-member content because content gating was only looking for `member.status === 'paid'` which doesn't take into consideration members on a "complimentary" plan.
- added front-end acceptance tests for member access to posts
- updated content-gating check to take comped members into consideration
- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- Fixture posts cover the key features and give users an introduction to how to use their site
- This all comes from the marketing and design teams to refresh the look and feel of Ghost and give users the best possible onboarding experience
Note: this fixture overhaul includes
- new content for new 4.0 features
- regenerated post content using our updated mobiledoc structure
- a switch from British to US English
refs 2bba9989db
- Note: this will require new fixtures so that the navigation links actually work
- These updates are all in aid of getting the best possible default setup and onboarding experinence for new Ghost users
- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- This all comes from the marketing and design teams to refresh the look and feel of Ghost
Note on accent color:
This commit changes the default accent colour again.
The intention is that new sites should get #FF1A75 (pink) as their default.
Any existing sites that do not have an accent colour set yet, should get #15171A (black) on upgrading to 4.0.
These are different as they are different experinces. Fresh sites will be guided to pick a color, so
a bright color is more visible and helps to see what can be done, whilst existing sites get a muted
black, that should be a sensible fall back color.
refs TryGhost/Team#535
We want to ensure that a site will always have a default value of
`'#15171A'` for the accent_color setting.
Since the boot process changed we have three cases to account for:
1. Setting does not exist
2. Setting exists with no value
3. Setting exists with a value
It is only in the case of 2. that we want the migration to update the
database with a default value.
In the case of 3. the site owner has already set a value, which we do
not want to override.
In the case of 1. the setting will be created (and populated with
default value) from the default-settings.json file, by the
populateDefaults method called from the settings service
We also update the accent_color setting to include a non-empty
validation, to ensure that the setting will always have a value, as
sites before 4.x may have an empty accent_color, we must update the
importer to set the default value if one is not present. Otherwise we
would run into validation errors and even if we didn't would have an
invalid database state.
no-issue
We were originally checking the state of the database, e.g. if a foreign
key constraint existed, so that we could conditionally act upon it. This
was to ensure that our migrations are idempotent.
Some database configurations, for example if you have many databases on
a single MySQL instance, would cause these information_schema queries to
take an exceptionally long time.
In order to speed up migrations, we instead attempt the action we want
to apply to the database, and then catch relevant errors to ensure the
migration is idempotent.
SQLite does not error when adding duplicate foreign or primary key
constraints, meaning that we must keep in pre-checks for these
operations, when running on SQLite
Co-authored-by: Daniel Lockyer <hi@daniellockyer.com>
no issue
Upgrading from v1 or v2 can result in successful upgrades but with mobiledoc errors showing in the logs:
```
NAME: InternalServerError
MESSAGE: Mobiledoc card 'card-markdown' not found.
```
The errors do not signify a problem as long as the 4.0 migrations run because those rename the deprecated card before re-rendering.
- `@tryghost/kg-default-cards` dropped support for `card-markdown` cards. 4.0 migrations handled this by renaming all `card-markdown` cards to `markdown` before re-generating any content
- 2.0 and 3.0 also had migrations that re-generated content but they are run before the 4.0 card rename migration meaning that the mobiledoc renderer sees cards that it doesn't know about. The behaviour for unknown cards is to log an error and skip rendering of that card
- by NOOPing the 2.0 and 3.0 migrations we eliminate the incompatibility errors and reduce the amount of processing the upgrade needs to perform
refs https://github.com/TryGhost/Ghost/pull/12736
refs https://github.com/TryGhost/Team/issues/467
knex's `parse()` method is only called on data when directly fetched from the db. This was causing problems when model instances are passed around via events for example because `.get('key')` will return data that was directly set on the model without having gone through the `parse()` transformations. The result of this inconsistency was settings appearing correct when Ghost started up but then being broken as soon as a setting was changed.
- moved absolute/relative->transform-ready URL transformations from the API input serializers to the model's `format()` method and replaced with a relative->absolute transform in API input serializers
- results in consistency because `.get()` on a settings model will always return an URL
- removed transform-ready->absolute transforms from the API output serializers as that is now handled at the model-layer
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Ghost/pull/12731
- settings are mostly fetched directly from the settings cache rather than via the API so they aren't subject to the API-level output serializers that transform URLs meaning that URLs in the front-end ended up with raw `__GHOST_URL__` replacement strings
- added images to the Settings model's `parse()` method so they are transformed immediately when fetching from the database
closes https://github.com/TryGhost/Team/issues/467
- switches to storing "transform-ready" URLs in the database
- transform-ready URLs contain a `__GHOST_URL__` placeholder that corresponds to the configured url that gives a few benefits
- much faster and less memory intensive output transformations through not needing to parse html or markdown - the transform can be achieved using a straightforward regex find+replace
- ability to change to/from or rename subdirectory without any manual updates to the database
- modified existing 4.0 url-transformation migration rather than adding another one and repeating the transformation on posts rows
refs https://github.com/TryGhost/Team/issues/525
We expect the member event tables to be large, and they contain what is
considered metadata. For this reason we do not want to include them in
the export of sites.
refs https://github.com/TryGhost/Team/issues/469
- Cleans up response format for mrr and volume stats endpoint to more consistent pattern
- Removes `unit` attribute for now as its not used
refs: https://github.com/TryGhost/Team/issues/510
- Changed to return the full hostSettings key, not just the billing URL
- We are introducing several more settings that are needed by Admin including limits
- Passing the whole object makes this much easier to reason about as Admin has the exact same config as the server
refs: https://github.com/TryGhost/Team/issues/510
- The current member limit was implemented as a member-specific concept
- The new limit service is much more generic, here we are swapping old for new
- The updated concept here is blocking all publishing, not just email sending, when a site is over its member limit
- To determine that we are publishing a post, we must be in the model layer. The code has been moved to the permissible function which makes sense as this is a permissions error that we are throwing
- I've left the extra check for email retries in, in case there is some loophole here (but we may wish to change it)
refs: https://github.com/TryGhost/Team/issues/510
- In the case that host config is provided, keep staff users within the limiti
- The definition of a staff user is a user with a role other than Contributor, and whose status is not inactive
- Contributors don't count
- Suspended (status inactive) users don't count
- Locked users DO count
- Invited users DO count
- You can't invite more staff users whilst there are pending invites
- You can't unsuspend a user, or change the role on a user in such a way as will take you over your limit
- You can't import staff users - all imported users are automatically set to Contributors
- As part of this work, we are changing the default Ghost user to a Contributor otherwise it uses up a staff user
Note: there is one known active bug with this commit.
- Assume you have one remaining user within your limit. You send an invite, this works.
- You cannot "resend" that invite, it will think you're sending a new invite and hit the limit
- You must "revoke" that invite first, and create a new one
- This bug exists because the resend function uses the add endpoint & does a delete+add, but this hits the permission check before the delete
refs https://github.com/TryGhost/Team/issues/510
- When the host config was introduced it was incorrectly introduced as host_settings instead of hostSettings
- All other Ghost config uses camelCase, so changing this now before it becomes a problem
- Note: Also removed some rogue return awaits that don't make sense. It's not possible to hit that case, but cleaning up anyway
refs https://github.com/TryGhost/Team/issues/509
- Allows to update and read 'locale' key along with the deprecated 'lang'
- In Ghost v5 the 'lang' key will be dropped and the migration in settings table will clean up the key name to match the one exposed through the APIs
refs https://github.com/TryGhost/Team/issues/509
- During 3.x we standardised on "lang" instead of "default_locale" for the site setting, which was an assumption based on an earlier change to @site.lang to make <html lang="@site.lang"> read nicer. This was a mistake as the field represents more than "lang" its a "locale". With this changeset we introduce a transition to use "locale" name for the value instead of "lang"
- Adds `@site.locale" value in as well as new 'locale' property in Content API's response
- "lang" will be considered as deprecated starting with API v4 and will be dropped completely with API v5
refs https://github.com/TryGhost/Team/issues/509
- Flipping around key/newKey pair allows to map multiple keys to the same field in the settings
- This becomes handy when there's a need to deprecate a field. For example, we are about to introduce a 'locale' setting which would need to map to 'lang' db key, with current structure it's impossible to have many:1 mapping because it's impossible to have duplicate keys in the JS object ("hash")
refs https://github.com/TryGhost/Team/issues/513
refs https://github.com/TryGhost/Team/issues/477
- We have skipped work to improve the 'defaultTo' value when working on Ghost 4.0 release, so adding this comment while context loaded
- defaultTo should not be set to anything as it leads to more maintenance work during major version bump
- having validation might make sense but could lead to similar maintenance work unless it's linked to some global nosion of "supported API versions" used everywhere
refs https://github.com/TryGhost/Team/issues/513
- Having exports in no particular order was making it hard to spot if there's anything missing
- Having a DESC order on the exported API versions makes it very easy to modify and maintain in the future
refs: https://github.com/TryGhost/Team/issues/510
- added and wired up the new limit service, which is a lazy-loaded service
- this handles the case that there are host limits set in config, and wraps all the logic needed for detecting exceeded limits & throwing limit errors
- expects limits to be set in config under `host_settings.limits`
- supported limits are managed in the limit service, outside of core
refs https://github.com/TryGhost/Ghost/issues/12610
refs https://github.com/mailgun/mailgun-js-boland/blob/v0.22.0/lib/request.js#L285-L333
The mailgun domain is used by the mailgun API to construct the URL for
the API. e.g for a domain of "mg.example.com" the URL for the API
messages would look like:
https://api.mailgun.net/v3/mg.example.com/messages
One weird thing about the mailgun API is that if the path does not map
to an API endpoint, then instead of a 404, we get a 200, with a body of
"Mailgun Magnificent API".
The `mailgun-js` library which we use, expects a JSON response, and will
return a body of undefined if it does not get one.
This all resulted in us trying to read the property `id` of an undefined
`body` variable. The fix here is to reject the containing Promise, if
there is no body. So that the default error handling will kick in.
refs https://github.com/TryGhost/Team/issues/513
- Bumping version as this is now the latest stable API.
- The change might cause this particular side effect (acceptable for major version): if a member requests a login URL when the instance is on 3.x version and site owner upgrades do 4.x before member authenticates through login link, the login will fail and the member will have to request a new login URL.
refs https://github.com/TryGhost/Ghost/pull/12541
refs https://github.com/TryGhost/Ghost/pull/12689
- the analytics job had been switched to create it's own instance of EmailAnalyticsService to avoid requiring logging but the analytics extraction branch was created before this change and wasn't picked up when merging
- pulled `queries` option object into a separate file for re-use
- updated `fetchLatest` job to conform to extracted library interface
closes https://github.com/TryGhost/Team/issues/493
- all functionality except that directly related to Ghost's database and business logic now lives in external packages
- @tryghost/email-analytics-service
- @tryghost/email-analytics-provider-mailgun
refs https://github.com/TryGhost/Ghost/issues/12716
- This change is made to avoid using now deprecated 'v2' API anywhere in the codebase.
- Switching to 'v4' should not cause sideeffects as this parameter is always present within the URL when Admin API is used
refs https://github.com/TryGhost/Ghost/issues/12716
- The code in serializePostModel was broken and always defaulted to 'v3'! It refered to non-existent `model.get('api_version')` there's no such field in posts model! Changed the implementation so that the API version is passed in as a parameter to the method instead
- The style of providing "defaults" everywhere creates a need for future maintenance when we bump the version e.g in Ghost v5. Maybe reworking these methods to require a passed version and throwing an error instead would be more maintainable long-term?
refs https://github.com/TryGhost/Ghost/issues/12716
- The default API in Ghost v4 will be `'v4'`. When a new webhook is created or any data passes through serialization it should assume `v4` as a fallback if not specified.
refs https://github.com/TryGhost/Team/issues/498
Requests to fetch the data of the logged in member made without a
session cookie were responsing with 400 Bad Request. This was incorrect
and always should have been a 401 Unauthorized.
refs https://github.com/TryGhost/Team/issues/498
This was filling up logs with less than useful information - for every
single request made by a non-member to the frontend. Be gone!
no-issue
The slack setting in 1.x did not have a `username` property. When
updating from 1.x, the migration to move the slack setting from a JSON
blob to two individual settings, would assume that the setting in the
database would have a value for the `username` property. This resulted
in errors on SQLite "sqlite does not support inserting default values."
The fix here is to add defaults when reading from the database, meaning
that we will _never_ attempt to insert `undefined`
refs https://github.com/TryGhost/Team/issues/476
* Moved paid subscription events population migration
This migration relies on the members_stripe_customers and
members_stripe_customer_subscriptions tables having no orphaned records
in order for it to correcly generate its data.
The migration to clean up orphaned records in those tables has not been
implemented yet, moving this migration free's up the "14" slot
* Removed orphaned stripe data from SQLite3
SQLite databases do not handle removing orphaned stripe records after a
member has been deleted. Our migration to populate the paid subscription
events relies on each customer and subscription being associated with a
member.
refs https://github.com/TryGhost/Team/issues/494
The migrations in 3.1.0 which added email permissions did not add those
permissions to the roles. This means that whilst we have the permissions
in the database, only the Owner role could use any of them.
This migration ensures that the email related permissions are added to
the correct roles.
refs https://github.com/TryGhost/Ghost/issues/12602
As part of the member events, we added a third status of 'comped'.
Members with a status of 'comped' should still be considered paid, so
this fixes the definition of the paid flag to take that into account.
refs 17feb14e4a
- The original commit adding this intended to add transactions, following the pattern of always forcing a transaction when we use bookshelf-relations
- (We use bookshelf-relations here because integrations have api-keys and webhooks associated wtih them and we upsert as one)
- These add and edit methods were inadvertently added to the wrong argument object/section of bookshelf (really fucking easily done, one day we will fix bookshelf so its easier to work with)
- Bottom line: these methods have never been called
- I tried moving them to the right section, but this created test failures throughout our acceptance tests:
- Error: Transaction query already complete, run with DEBUG=knex:tx for more info
- This is likely because we need to account for integrations being used as part of the auth step in the before part of tests
- In terms of yak-shaving, fixing these tests is one step too far right now. I think not having this code here at all is a better state than having it look like it works when it doesn't
no issue
- the analytice-node v4 update contains a breaking change that throws an
exception if the message is over 32kb
- I'm pretty sure we won't hit this, but it's good to track these errors
anyway and stop Ghost from bombing out if there's an issue
- this commit wraps the tracking call and adds Sentry + logging to the
error
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we've introduced backwards-compatible changes to rendering in 3.0 such as srcset and sizes which will only have taken effect on posts created or edited since the changes were made
- 4.0 brings additional changes such as image card width/height
- re-generating the `html` field of all posts from the `mobiledoc` brings all content up to latest rendering output
refs https://github.com/TryGhost/Team/issues/469
The initial implementation was just to get us going with the dashboard
but the requirements have changed now, rather than updating the code we
allow to pass the `limit` options so the Admin can choose how many
events to display.
- There is now one true way to start Ghost - you create a server, and then call start with an express app
- We may well expand this again to improve testing pathways in future, but it will be done with a bit more clarity about expectations
refs https://github.com/TryGhost/Team/issues/221#issuecomment-759105424
- Mailgun responds to an email send with a provider id in the format `<x@y.com>` but everywhere else it's used in their API it uses the format `x@y.com`
- updates email batch save to strip the brackets, and migration removes brackets from existing records so we no longer have to add special handling for the stored id any time we use it
refs https://github.com/TryGhost/Ghost/issues/12496
- having the logging require here means that workers wanting to use the db are unable to do so without requiring logging as a side-effect
- `connection.loggingHook` does not appear to be widely used for anything outside of specific debugging scenarios when using MySQL so it should be safe to disable until a proper fix is found for workers+logging leaking file descriptors
refs https://github.com/TryGhost/Ghost/issues/12496
By requiring the models layer the shared logging util was being required as a side-effect causing the open file descriptors problem to continue. Removing logging from the models layer isn't feasible due to deep require chains spreading across the codebase, it's much quicker to remove the need for models in the analytics job.
- models layer was only needed because it's used by the session service
- updated analytics job to create it's own instance of `EmailAnalyticsService` rather than the default instance in order to pass in custom dependencies
- pass in custom `logging` object that uses `parentPort.postMessage` as a way of writing log output
- pass in custom `settings` object that returns settings that have been manually fetched and cached during job instantiation
refs https://github.com/TryGhost/Ghost/issues/12633
Adds new `browse` endpoint for emails that allows Admin to check performance of newsletters over time and show stats on dashboard as primary usecase
refs https://github.com/TryGhost/Ghost/issues/12496
- Handling logging in the main thread avoids file handle leaks which happen due to leaky implementation of bunyan logger (see referenced issue for more context)
- Bumped job-manager version to allow for `workerMessageHandler` callback funciton
refs https://github.com/TryGhost/Ghost/issues/12496
- Using ghost-ignition logging caused file handle leaks. As there is no straight-forward way to handle write streams with bunyan (ghost-ignition's underlying logging library) this method of logging was chosen as an alternative to keep the amount of open file handles to minimum
- The follow up changes will include custom formatter for jobs service which should make logging match the same format as has been used inside the jobs
refs https://github.com/TryGhost/Ghost/pull/12598
- This changeset adds idepmotence to situations where unique contraint has to be dropped or added to the table
- Note '4.0/07-alter-unique-constraint-for-posts-slug.js` was migration that was effected by lack of idempotence
refs https://github.com/TryGhost/Ghost/issues/10318
- Storing JSON object in settings has caused multiple bugs in the past and was considered an antipattern
- This is a last bit in the long process of getting rid of "object" settings in the database. At this point there should be no settings with this type. Yey!
- In the old boot the server wasn't started til we were ready
- In new boot, we start the server immediately and send the old started event
- Then, when we are ready to accept some traffic, we send a ready event
- At the moment, ready isn't quite sent at the right time:
- It _should_ be when we're ready to serve real traffic, not just send 503s
- This is after the URL generation has finished
- But this requires more refactoring work :(
- So for now we send when everything else is ready
- This really needs some tests
- Notify is a more familiar name e.g. systemd has the sd_notify system which this is similar to
- We're actually announcing the server started, it's not actually ready for traffic (will serve 503s)
- use theme.ready for loading themes instead of server.start and properly clean this up
- remove server.start and server.stop as they are no longer used (only server.start was used, and only for themes)
- we're moving away from the pattern of using global events like this as they are hard to reason about
- This commit removes the old boot process and any files that are no longer needed as a result
- Remove the duplicate event for triggering inactive themes to load
- Tidied up a few other bits
no issue
- all migrations should contain a logging message
- info for successful things, or warn if we deviated from what was
expected for some reason
- also added some spacing to make them easier to read
closes https://github.com/TryGhost/Team/issues/468
- updated post-gating
- clears excerpt if there's no access
- rebuilds excerpt from free preview if paywall card is used and there's no custom excerpt
closes https://github.com/TryGhost/Team/issues/466
- upgraded kg-default-cards to include paywall card
- extracted `htmlToPlaintext` from post model to shared util for re-use
- updated post-gating to set html+plaintext to the free preview if a paywall card has been used
- re-generates plaintext from the truncated html using `htmlToPlaintext` util
- display free content in the `{{content}}` helper via the default CTA template
refs https://github.com/TryGhost/Ghost/issues/10318
- API changes introduced:
canary/v4 Admin API
GET /settings/ (browse)
+ "unsplash" present in response as boolean value
GET /settings/:settingName (read)
+ "unsplash" present in response as boolean value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts ONLY boolean format
v3 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts either boolean or object formats
v2 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts object format
refs https://github.com/TryGhost/Ghost/pull/12631
- Initialy writen migration would break if executed twice
- Also accounted for situation if only part of the migration executed leaving only one of the values migrated
- Allow the frontend to accept post messages to generate previews of the frontend
- Created a new endpoint in admin we can use to render these previews, which is possibly not necessary
- Supports a limited group of settings, which can easily be expanded, but care should be taken if expanding to use user-provided strings
refs https://github.com/TryGhost/Ghost/issues/10318
- Because members is effectively "enabled" by default starting Ghost 4.0 have hardcoded labs setting to be such. The alternative of removing this key from labs would be equivalent to `labs.members === false` which is undesireable and would mean additional work on theme developer's side.
refs https://github.com/TryGhost/Ghost/issues/10318
- `labs` setting is dropped from setting values as the use of JSON objec
to sore settings has been deprecated
- `labs` setting is no longer accepted as a paramter in the Settings API nor the
impoprter. The value is ignored if present in the POST/PUT requests and
returns 404 in case it is requested by key at `GET /settings/:key`
refs https://github.com/TryGhost/Team/issues/332
- The last value that has been used in the code was "members"
- By default members will be always "on" starting Ghost 4.0, so there's no need for this flag anymore
- Therefore there's no real need to keep "labs" around
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we added robust url transformation in Ghost 3.0 which ensures newly created or edited content is stored with relative URLs in the database
- this migration loops over all posts (and posts_meta) in the database performing the absolute->relative transformation to normalise data for old posts
closes https://github.com/TryGhost/Ghost#12562
- From NodeJS v13 `error.errno` returns error code instead of a string. Because of that use friendly "port is already in use" message did not work anymore.
- Changed to use `error.code` which acts the same way as `error.errno` in older NodeJS versions.
refs https://github.com/TryGhost/Ghost/issues/12646
- adds `ghostVersion` property to mobiledoc of all posts so that we can pin the rendering output of existing content to protect against future breaking changes
- cleans any deprecated markdown card names from old Ghost 1.0 content
- cleans up HR cards that had an [accidental payload added](610d801bcf)
refs https://github.com/TryGhost/Ghost/issues/12646
- if the version is missing then the content was created before Ghost 4.0
- setting the version to `'3.0'` means it will continue to use the same rendering output so there are no unexpected breaking changes when migrating content
refs https://github.com/TryGhost/Ghost/issues/12602
* Updated members_status_events table
By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range
* Updated MEGA to handle addition of 'comped' status
With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"
* Updated customQuery for MemberStatusEvent
With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.
* Populated members_status_events table
As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.
The assumptions used in backfilling is that a Member's current status,
is their only status.
refs https://github.com/TryGhost/Ghost/issues/12602
As part of collecting Member event data, we have added a third status
for members "comped" - this fixes the population of the column to handle
this
refs https://github.com/TryGhost/Ghost/issues/12602
The gross volume aggregate query was incorrectly naming the selected `SUM(amount)` value as `gross_volume` instead of expected `volume_delta`
refs https://github.com/TryGhost/Ghost/issues/12633
Adds new stats endpoints for showing member dashboard, which fetches aggregate date over member events tables to form relevant datasets -
- Adds new endpoint for tracking MRR of site over time - `/members/stats/mrr`
- Adds new endpoint for tracking subscribers on site over time - `/members/stats/subscribers`
- Adds new endpoint for tracking gross volume of site over time - `/members/stats/gross_volume`
- Adds new endpoint for tracking member count on site over time - `/members/stats/count`
refs https://github.com/TryGhost/Ghost/issues/12646
- when a blank mobiledoc document is created, ensure the correct Ghost version is added so content output is pinned to the version of Ghost that created it
- Currently the new boot process breaks the loading of inactive themes because the theme loader event isn't wired until after the server.start event is fired
- This is a WIP fix, there needs to be some bigger refactoring of the boot process once the old process is gone
refs https://github.com/TryGhost/Ghost/issues/12646
- `card-markdown` support is being removed from the default mobiledoc cards so we need to make sure imported content doesn't fail
- removed long-unused `cardName` payload property from tests
refs https://github.com/TryGhost/Ghost/issues/10318
- Object format used in previous "slack" setting was considered an
anti-pattern. Flag structure of separate slack_url and slack_username
values was extracted out of the "slack" JSON.
refs https://github.com/TryGhost/Ghost/issues/12602
As we only want to add events for when an email is changed, we need to store the previous email if we want a complete log of all emails for a member
refs https://github.com/TryGhost/Ghost/issues/12608
- adds `admin/canary/themes/install` endpoint to the Admin API
- requires two query params. `source` must be set to "github". `ref` should refer to a GitHub repo in the format "{org}/{repo}"
- downloads zip archive for the repo from github
- runs downloaded zip through the same process as uploaded zips
refs https://github.com/TryGhost/Ghost/issues/12602
* Added members_payment_events table
This table will store successful and unsuccessful payment attempts, and
can be used to calculate gross volume over time.
* Added members_login_events table
This table can be used to audit member logins
* Added members_email_change_events table
This table will allow us to store a history of email addresses associated with a member
* Added members_status_events table
This table will allow us to track the change in status over time for members, as well
as calculate aggregates over time, e.g. paid members over time
* Added members_paid_subscription_events
This table will allow us to track subscriptions changes for members, as well as
calculating MRR over time
refs https://github.com/TryGhost/Ghost/issues/12567
- Changing unique constraint from slug to slug+type should allow for posts and pages to be created with the same slug
- The constraint will be present on application layer for API v4 while we figure out how to deal with it in API v5
Background:
- Ghosts existing boot process is split across multiple files, has affordances for outdated ways of running Ghost and is generally non-linear making it nigh-impossible to follow
- The web of dependencies that are loaded on boot are also impossible to unpick, which makes it really hard to decouple Ghost
- With 4.0 we want to introduce a new, linear, simpler, clearer way to boot up Ghost to unlock decoupling Ghost into much smaller pieces
This commit:
- adds a new ghost.js file which switches between boot mode with `node index` or `node index old` so that if we find bugs we can work around them this week
- Note: the old boot process will go away very soon, but ghost.js will remain as the interface between the command to start Ghost and the application code
- reworks the database migration process into a standalone utility, so that the DB is handled as one simple step of the boot process, decoupled from everything else
- is missing tests for this new db utility
- leaves a lot of work to do around loading core code, services, express apps in a sensible order, as work to fix this would start to break the old boot process
- doesn't use the new maintenance app because we aren't restarting the server here, instead we have the concept of a "core app" that starts in maintenance mode - need to think about how apps will be decoupled in the near future
closes https://github.com/TryGhost/Ghost/issues/12600
The bookshelf-relations plugin which we use will **remove** all
relations when they are set to an empty array, but will leave them alone
if it's set to undefined.
Our logic to deduplicate uppercase & lowercase version of the same label
was in advertently always setting the labels to an array, but when the
model was saved without passing the labels, this array would be empty.
Here we've added a check which will skip all label handling, if there
are no labels set.
refs https://github.com/TryGhost/Ghost/issues/12602
In order to build up a list of un/subscribes over time we have to use
the customQuery functionality to run SQL aggregates - this is "hidden"
behind an option, so that we can find{All,Page} as usual.
refs https://github.com/TryGhost/Ghost/issues/12602
Member Events in general are read-only after they've been created, so
we've explicitly disallowed destroying and editing via the model
refs https://github.com/TryGhost/Ghost/issues/12602
This allows us to backfill the data for testing purposes as well as for
use by existing sites. The mechanism for determining the date of an
unsubscribe event is crude, we use the date of the last update, which we
might expect to be _sometimes_ correct, but not always. We store the
`source` as NULL for unsubscribes as we cannot determine that with any
certainty whatsoever.
refs https://github.com/TryGhost/Ghost/issues/12567
- The method was using unneeded lodash dependency and was too complex for what it was doing
- Reshuffled internal code to use native JS filter/forEach iterators
refs https://github.com/TryGhost/Ghost/issues/12567
- Introduced here @@UNIQUE_CONSTRAINTS@@ notation allows to create unique contraints over multiple database fields. This will be needed to change posts' table unique constraint from `slug` to `slug+type`.
- The notation is equivalent to SQL's: UNIQUE(column_name1,column_name2)
- Example use in schema:
posts: {
slug: {type: 'string', maxlength: 191, nullable: false},
type: {type: 'string', maxlength: 50, nullable: false, defaultTo: 'post', validations: {isIn: [['post', 'page']]}},
'@@UNIQUES@@': [
['slug', 'type']
]
}
no issue
- when saving Stripe keys with `stripeDirect: true` config, if either key didn't match the key format the returned validation error always contained `stripe_secret_key`
- updated to output `stripe_publishable_key` if it was the publishable key that was invalid
refs https://github.com/TryGhost/Ghost/issues/12565
- Dropping tables happens sporadicaly, usually during major version releases. It made sense to create a utility based on previous migrations of this type (e.g.: 3.0/03-drop-client-auth-tables.js migration) to avoid code duplication in the future
no-issue
This removes all references to the members labs setting, any code that was run conditionally behind this flag now runs unconditionally.
* Removed usage of Members labs flag
* Removed tests for Members disabled
* Added dynamic keypair generation for when setting is missing
no refs
- Updates member model serializer to directly set subscriptions on member object instead of `stripe.subscriptions`
- Updates all references to members subscriptions from nested `stripe.subscriptions` to `subscriptions`
- Updates v3 API serializer to still use `stripe.subscriptions`
- Updates tests
no-issue
* Removed support for paid param from v3 & canary API
* Updated active subscription checks to use status flag
* Updated MEGA to use status filter over paid flag
* Removed support for paid option at model level
* Installed @tryghost/members-api@1.0.0-rc.0
* Updated members fixtures
no-issue
This value is nullable from the Stripe API so we're making sure that we
can store exact values locally.
SQLite3 does not supports altering columns so instead we have to:
1. Create a temp table and copy the data to it
2. Delete the original table
3. Recreate the original table with the necessary modifications
4. Copy the data from the temp table
5. Drop the temp table
refs #12568
- DI pattern through constructor allows for very clear separation of concerns and makes it easy to extract the class into a separate module
refs #12568
- When the server is in maintenance mode (e.g. during migration) it needs to serve a 503 page. Previously this role was delegated to the "frontend" which tightly coupled server bootup and frontend.
- With a dedicated HTTP application serving up 503 server is no longer coupled to the frontend during maintenance
refs #12568
- Allowing express app (frontend) to be booted after the migrations
decouples server maintenance mode from a need to have the "frontend"
loaded beforehand
refs #12160
This flag will allow us easier filtering of members via the API
* Added status column to members table
This flag will be used to determine if a member is free or paid, rather
than relying on joins with the customers and subscriptions tables.
* Added migration to populate members.status
As we add the column with a default value of "free" we only need to care
about the paid members here. We also preemptively handle migrations for
SQLite where there are > 998 paid members.
refs #12568
- The readivility of multiple nested promises is very poor
- Refactored this piece of code before doing a bigger reshuffle of frontend/server boot process
no issue
We added `portal-action` and `requestSrc` in 3.x to allow Portal to handle notifications only for auth actions trigged while using it directly, so that existing themes are not affected in any way. Going forward in 4.0, we don't want to have any special handling in backend for Portal but instead expect themes to handle any Portal specific behavior directly.
- Removes setting of `portal-action` for auth actions like signup
- Removes `requestSrc` being passed through to determine portal actions
closes https://github.com/TryGhost/Ghost/issues/12449
We’re starting to bump into errors with our current exporter due to the size of some of the tables in the db and hitting an issue with Ghost running out of memory during export. The intention for the export/import is not to be backup/restore functionality, but for exporting content and authors.
In addition, exporting and re-importing Stripe secret/publishable keys can cause unexpected side-effects for sites that can has major side-effects. This change -
- Removes `email_batches` and `email_recipients` tables from export data to reduce export size due to large amount of analytics data
- Removes stripe's secret/publishable/webhook keys to avoid unexpected issues with re-import
closes https://github.com/TryGhost/Ghost/issues/12347
This change allows a token to be used multiple times for the first 10
seconds after its initial use, this will stop dynamic link checking
software from invaliding magic links.
no refs
Adds new FirstPromoter settings similar to amp, which allows sites to take advantage of FirstPromoter to launch their own member referral program natively.
- Adds new firstpromoter settings group
- Adds `firstpromoter` setting to group
- Adds `firstpromoter_id` setting to group for FirstPromoter referral tracking id
- Updated tests
refs #12537
- Refactored overuse of rewire mocking blog-version.
- Fixed a bug introduced along the way when duplicate notifications errored instead of returning empty result
refs #12537
- notifications controllers were overbloated with non controller related code and were identical. It is important to reduce unnecessary code ahead of v4 API introduction
- Follow up commit will transform newly created module into a class following DI pattern
refs #12537
- Class syntax is preferred over module functions because of constructor parameter injection (DI) which allows for easier module decoupling and testing
refs #12537
- `stats` method in members controller is quite big and does much more then controller method code should - few calls to relevant modules
- Extracted code "as is" into members serivce
- Next step will be to refactor this module as a class pattern with DI parameters
refs https://github.com/TryGhost/Team/issues/446
Mailgun permanent failure events do not always correspond to unsubscribe-level events as originally thought, meaning some members could be unsubscribed unexpectedly due to delivery hiccups.
- disabled auto-unsubscribe on permanent failure events in the analytics event processor
- list maintenance will be added back in the future via alternative means
closes https://github.com/TryGhost/Ghost/issues/12493
- bumps `bookshelf-relations` to new version that allows for `hasMany` children to be kept when a parent record is destroyed
- adds necessary config to the `Member.email_recipients` relationship to keep related records when a member is deleted
closes https://github.com/TryGhost/Ghost/issues/12492
The changes to email processing models had set replyTo address for an email batch as `reply_to` instead of `replyTo` which was not picked by mailgun service for setting newsletter reply address
closes#12503
Since we include _all_ subscriptions in the `stripeSubscriptions`
relation, we must check that for an existing active complimentary
subscription to decide whether or not we should skip creation of a
complimentary subscription
ref #10898
- The redirects configuration's `to` & `from` URL parameters used to ignore it's query string parameters, which resulted in unexpected behavior
- Current changeset only partially fixes the issue. Now `to` URL's query parameters always take precedence over incoming query parameters and the rest of query parameters are passed through.
closes#12181
refs https://github.com/TryGhost/Ghost/pull/12265
- The 'core' value is invalid for settings 'type' column
- The 'core' default value for 'type' column in model is also invalid
- Both need to be removed as they are never used and only introduce confusion into the codebase
no issue
- Previously used notation was not a valid JSDoc one. With this correction IDEs will provide more accurate information about all furnction parameters
refs https://github.com/TryGhost/Ghost-Utils/issues/118
- Duplicating error handling across jobs is not best developer experience. Also, having custom error handling logic did not allow for recommended worker script behavior: allowing for unhandled exceptions to bubble up and be managed by parent process
refs c1d66f0b01
- fixed base model allowing '@@INDEXES@@' as a permitted attribute/order
- fixed base model automatically setting `@@INDEXES@@` to null on the model when creating
- added `doAuth('members:emails')`
- creates an `email_batch` record attached to the first email in the fixtures
- creates an `email_recipients` record for each member
- runs analytics aggregation so the email and member counts are as expected
- added acceptance test for `/member/:id/?include=email_recipients`
no-issue
We now include all subscriptions as part of the member, so we need to
ensure the comped flag is only true if the member has an **active**
complimentary plan
refs https://github.com/TryGhost/Ghost-Admin/pull/1796
We want to be able to display an email activity timeline in Ghost-Admin for each member. The quickest way to achieve that right now is to provide access to the `email_recipient` data for the member when fetching, this will allow clients to build up a timeline based on the event timestamps included with each email_recipient/email pair.
- sets up `email_recipients` relationship in `Member` model
- updates members API read endpoint to accept an `email_recipients` include parameter
- appends `email_recipients.email` to the `withRelated` array when `email_recipients` is included so that we have data available for email subject and html/plaintext for previews
- updates members API output serializer to include the email_recipients object in the output
no issue
- sqlite will store a float in an integer column due to it's type affinity resulting in long decimal numbers in the UI when we're expecting an integer
- use the `ROUND()` function to ensure we're storing integers in place of floats when performing open rate average calculations
no-issue
This module encapsulates the work around performing imports, it
currently uses the concept of a "Job" which at the moment is not
persisted to the database, however when we want to look at resuming
imports after a server restart, this should give us the flexibility to
do it.
refs https://github.com/TryGhost/Ghost/issues/12461
- added two default aggregations for overall email count and opened email count
- when number of tracked emails is sufficient add the open rate aggregation to the update query
refs https://github.com/TryGhost/Ghost/issues/12461
- adds `members.email_count` and `members.email_opened_count` columns to contain cached counts for faster queries when outputting member data via API
- adds migration to populate cached counts with existing data
- tested locally on ~50k members which took ~4sec on mysql
- updates members output serializer to include the new fields in API output
refs https://github.com/TryGhost/Ghost/issues/12421
- add `orderRawQuery` function to members model so that we can ensure members with an open rate are ordered before members without an open rate no matter the order direction chosen
- added `email_open_rate` to members in the test fixtures to allow testing of order
refs https://github.com/TryGhost/Ghost/issues/12421
requires https://github.com/TryGhost/Ghost/pull/12457
- updates stats aggregator to calculate and store an open rate for each member
- uses two queries because I couldn't find a reasonable approach to perform the update in a single query as per the email aggregation
- benchmarked locally at <1sec/1000members
- will not store an open rate unless the number of tracked emails sent to a member is above a certain threshold (defaults to 5) to avoid new members being heavily weighted
- fixes typo in EmailAnalytics that was stopping member stats from being aggregated
refs https://github.com/TryGhost/Ghost/issues/12421
- nullable so we can distinguish between members that have and haven't received any trackable emails
- indexed because we'll be using this column for sorting
refs https://github.com/TryGhost/Ghost/issues/12420
- updated `order` bookshelf plugin's `parseOrderOption()` method to return multiple order-related properties
- `order` same as before, a key-value object of property-direction
- `orderRaw` new property that is a raw SQL order string generated from `orderRawQuery()` method in models
- `eagerLoad` new property that is an array of properties the `eagerLoad` plugin should use to join across
- updated `pagination.fetchAll()` to apply normal order + raw order if both are available and to handle eager loading / joins when `options.eagerLoad` is populated
- updated post model to include details for email relationship and to add `orderRawQuery()` that allows `email.open_rate` to be used as an order option
no issue
- job registration was checking for submitted emails in it's email count but the job registration method is called as soon as an email is created meaning the email has a status of 'pending' which prevented the analytics job from being started until a second email was sent
no issue
- email analytics may be desirable to fully switch off in certain circumstances, when that happens we want to prevent related background jobs from running and expose the feature flag via the config endpoint in the Admin API so that clients can adjust accordingly
no issue
- if emails are older than 30 days we wouldn't be able to fetch any analytics for them and if a site used emails in the past but is no longer using them it doesn't make sense to keep potentially expensive background worker threads spinning up
no issue
- recurring jobs spin up worker threads which can be quite CPU intensive even when not performing much processing, this can be problematic in environments where there are many Ghost instances running
- updated the email job scheduling to be skipped on bootup when there are no emails in the database and to be started when the first email is created as long as we're not in testing env
- increase analytics job schedule from every 2 minutes to every 5 minutes to help spread the load further across instances
no issue
- typically cron/later schedules will schedule for :00 on the minute which would create API spikes with every members-email-using Ghost site hitting the API at the same time
- adjusted the scheduling to use cron syntax with job runs every 2 minutes on 1,3,5... or 2,4,6... and a random seconds value to smooth usage across sites
no issue
- it's possible background jobs may cause unintended side-effects so it's useful to have a kill-switch to disable them individually to keep sites working
- users imported from CSV with no created_at date where having their created_at date being stored as an int rather than a datetime.
- this was causing parsing issues with the graph so this commit fixes the formatting
closes#12083
- fixes a parsing issue where negative offset values were incorrectly having the + sign added regardless of actual offset for sqlite databases.
- for mysql databases absolute values of offset were taken with sign applied where appropriate to stop issues where both hours and minutes could be negative which would cause both an issue with offsets that could present as -2:30 and by the look of the code also trigger extra padding to result in -2:-030 rather than the expected -2:30
no issue
- the 4.2.0 version of `sqlite3` that we're using is not compatible with `worker_threads`
- 5.0.0 should add support this but there are other errors
- 5.0.1 is released but not published (https://github.com/mapbox/node-sqlite3/issues/1386)
no issue
- added `EmailAnalyticsService`
- `.fetchAll()` grabs and processes all available events
- `.fetchLatest()` grabs and processes all events since the last seen event timestamp
- `EventProcessor` passed event objects and updates `email_recipients` or `members` records depending on the event being analytics or list hygiene
- always returns a `EventProcessingResult` instance so that progress can be tracked and merged across individual events, batches (pages of events), and total runs
- adds email_id and member_id to the returned result where appropriate so that the stats aggregator can limit processing to data that has changed
- sets `email_recipients.{delivered_at, opened_at, failed_at}` for analytics events
- sets `members.subscribed = false` for permanent failure/unsubscribed/complained list hygiene events
- `StatsAggregator` takes an `EventProcessingResult`-like object containing arrays of email ids and member ids on which to aggregate statistics.
- jobs for `fetch-latest` and `fetch-all` ready for use with the JobsService
- added `initialiseRecurringJobs()` function to Ghost bootup procedure that schedules the email analytics "fetch latest" job to run every minute
no issue
- cleans up unused tables `emails.{meta,stats}`
- adds timestamp columns `email_recipients.{delivered_at,opened_at,failed_at}` that can be used for event timelines and basic stats aggregation
- indexed because we want to sort by these columns to find the "latest event" when limiting Mailgun events API requests
- adds aggregated stats columns `emails.{delivered_count,opened_count,failed_count}`
- adds a composite index on `email_recipients.[email_id,member_email]` to dramatically speed up `email_recipient` update queries when processing events
- modifies the db initialisation to support an `'@@INDEXES@@'` key in table schema definition for composite indexes
closes https://github.com/TryGhost/Ghost/issues/12416
This fixes compatibility for the `send_email_when_published` option for
the Posts API.
The model layer only allows setting the `email_recipient_filter` column
when the `status` is being changed. Because of this we need to withhold
the `status` change until after we have determined the
`email_recipient_filter`.
- this helps simplify the code and gets rid of Promise chaining
- apparently I can't easily use an async function within filter, so I've
left it for now
refs https://github.com/TryGhost/Ghost/issues/12256
We no longer want to filter out cancelled subscriptions, so we are able
to remove the whereIn clause of the relation.
* Fixed paid flag on member
* Fixed content gating for members
Now that the subscriptions for a member include all of them, we must
explicitly check that the member has an active subscription in order to
consider them "paid"
- this helps bring all the code together so we can extract it in the
future
- turning it into a class also lets us easily inject the i18n instance
and store it locally
no issue
- the `'bulk-email`' tag was only being added to bulk emails if another more specific tag was set up via config
- we always want the `'bulk-email'` tag to be present for better event filtering
refs #12402
- With bumped version of job-manager it offloads job procesing into separate worker thread. Having jobs run out of main Ghost process even loop allows for safe job execution, which does not block Ghost from serving requests or performing other functions without a delay
- Added experimental data access to 'testmode' jobs. This should serve as an illustration of how to access data from the job layer
refs https://github.com/TryGhost/Ghost/issues/10354
- The serializer wasn't hooked up properly during the implementation (1a4497fc9a). It is not possible to hook it back in now as that would introduce breaking change to now stable v2 API.
- v3 (canary) has is properly hooked in, so there should not be a problem going forward
refs #12366
This implements redirection based on the settings for successful member sign up!
- Removes support for redirecting to `req.path` afterwards, this was never used and
we now have a more configurable implementation.
- Retains redirection to the homepage for unsuccessful sign up (invalid/expired token)
no-issue
This logic would assume that the option was always passed at the point
of publishing the post, which is not the case for scheduled posts.
Instead of setting the property to 'none' when the option is not
present, we take the approach of ONLY setting the propery when
1. It is present and not 'none'
2. The post is being published or scheduled
This means that scheduled posts will have the property set correctly,
and any future publishing will leave the it in the original state
no refs
- Migrations were using previous default for badge which was set to `false`
- Default config for badge was `false` which caused migration to switch off the badge, updated to `true`
- these helpers remove a lot of the duplicated code that we had when doing up/down column migrations, and provides a much shorter way of doing this in the future
no issue
Updated newsletter badge to be shown by default which was previously set to be hidden. Users can still control the visibility via Email settings in Ghost-Admin
no-issue
We want to give users to ability to customise the content of their newsletter, and the first step
toward that is a setting in which we can store text or html to embed in the template
no-issue
* Handled send_email_when_published in Posts API
This restores backwards compatibility of the Posts API allowing existing
clients to continue to use the `send_email_when_published` flag. This
change uses two edits, which is unfortunate. The reason being is that
this is an API compatibility issue, not a model issue, so we shouldn't
introduce code to the model layer to handle it. The visibility property
of the model is used to determine how to fall back, and because it can
be left out of the API request, and relies on a default in the settings,
we require that the model decide on the `visibility` before we run our
fallback logic (or we duplicate the `visibility` default at the cost of
maintenance in the future)
* Dropped send_email_when_published column from posts
Since this column is not used any more, we can drop it from the table.
We include an extra migration to repopulate the column in the event of
a rollback
* Updated importer to handle send_email_when_published
Because we currently export this value from Ghost, we should correctly
import it. This follows the same logic as the migrations for this value.
* Included send_email_when_published in API response
As our v3 API documentation includes `send_email_when_published` we must
retain backward compatibility by calculating the property.
* Fixed fields filter with send_email_when_published
* Added safety checks to frame properties
Some parts of the code pass a manually created "frame" which is missing
lots of properties, so we check for the existence of all of them before
using them.
* Fixed 3.1 migration to include columnDefinition
We require that migrations have all the information they need contained
within them as they run in an unknown state of the codebase, which could
be from the commit they are introduced, to any future commit. In this
case the column definition is removed from the schema in 3.38 and the
migration would fail when run in this version or later.
refs https://github.com/TryGhost/Ghost/issues/12355
- Adds new default settings for newsletter customisations - `newsletter_show_badge`, `newsletter_show_header` and `newsletter_body_font_category`
- Adds migrations to update group for new settings
- Add migration to update settings based on existing config value for newsletter settings
- Passes new newsletter settings to newsletter template and updates design based on them
- Fix tests
no issue
- These changes should allow easy testing of scheduled jobs and provide an implementation example for future jobs within Ghost
- Added experimental support to run jobs based on functions defined in modules
- Extendeded testmode "schedule" endpoint with an optional job name parameter which picks up job from "jobs" folder
- Adjusted scheduleJob method use and passed "schedule" as first parameter
- Bumped job-manager version to allow for all new functionality
no issue
- Living in separate folder will allow adding more testmode specific modules (for example test jobs)
- Mimicks the folder structure of how v2/canary routes are structured better
no-issue
* Used email_recipient_filter in MEGA
This officially decouples the newsletter recipients from the post
visibility allowing us to send emails to free members only
* Supported enum for send_email_when_published in model
This allows us to migrate from the previously used boolean to an enum
when we eventually rename the email_recipient_filter column to
send_email_when_published
* Updated the posts API to handle email_recipient_filter
We now no longer rely on the send_email_when_published property to send
newsletters, meaning we can remove the column and start cleaning up the
new columns name
* Handled draft status changes when emails not sent
We want to reset any concept of sending an email when a post is
transition to the draft status, if and only if, and email has not
already been sent. If an email has been sent, we should leave the email
related fields as they were.
* Removed send_email_when_published from add method
This is not supported at the model layer
* Removed email_recipient_filter from v2&Content API
This should not be exposed on previous api versions, or publicly
* Removed reference to send_email_when_published
This allows us to move completely to the email_recipient_filter
property, keeping the code clean and allowing us to delete the
send_email_when_published column in the database. We plan to then
migrate _back_ to the send_email_when_published name at both the
database and api level.
no issue
- set `emails.track_opens` to `true` when the `enableDeveloperExperiments` flag is set
- update mailgun bulk-email provider to pass the open-tracking header to Mailgun when the email's `track_opens` flag is set
- this check should never really be hit unless the DB has missing data,
but the code tries to read from an undefined variable and crashes, which we
can protect against
- this can be triggered by doing a `knex-migrator init` followed by a
`knex-migrator rollback --v 3.0 --force`
- 504509bb6 removed the global override for Promise
- there are a bunch of places in code that use Bluebird Promise methods,
but Bluebird wasn't being imported in these places
- this would have thrown errors all over the place
no issue
- tracking of bulk email opens can be enabled/disabled over time, if we're calculating analytics for emails we don't want emails which didn't have tracking enabled skewing the results so we need a record of whether tracking was enabled for each email
no issue
- We need an ability to early test scheduled jobs to refine the new feature and it's API. Should be used with caution
- To schedule an example scheduled job every 30 seconds run following request: `curl http://localhost:2368/ghost/api/schedule/every%2030%20seconds`
refs #11085
- Incorrect usage error was logged to the output when there was no recirecst configuration file present in the system. Previously an empty string was returned in such situation, resulting in "ENOENT" error, which was ignored through special handling.
- The fix resembles logic in redirects async getter function where empty array is returned when the config file does not exits.
- Attempting to read unexistent config should not ever happen and will be handled on the config service layer, this is why special "ENOENT" handling has been removed
no-issue
This column will allow us to store the canonical recipient filter on the
email resource giving us a detailed log of which members an email was
intended for
no-issue
This column will allow us to decouple the recipients of newsletter from
the `visibility` of a post, allowing us to send emails to specifically
free members, or to send paid posts as newsletters to all members.
closes#11085
- Ghost has been using YAML format for other configurations (e.g. routes). The plan is to move to this format for all user-edited settings files. By default JSON format is still used in Ghost Admin API v2/v3, but will be changed to YAML in API v4. Check referenced issue for more context.
- New format supports all the features available before. The main noticeable change is the structure of config file. It is now grouped by redirect HTTP code instead of specifying `"permanent": true | false` attribute for each config property. Example format for YAML config:
```
302:
/from-url/: /to-url/
301:
/category/([a-z0-9\-]+)/i: /tag/$1/
/v([0-9\.]+)/docs/([a-z0-9\-]+)/i: /docs/$2/
```
- Added 2 new endpoints: `POST redirects/upload` and `GET redirects/download`. These serve as an alias to current GET/POST `/redirects/json. "upload/download" naming pattern is introduced to match the convention with other resources that can be uploaded and downloaded (images, themes etc.). `/redirects/json` endpoints will be removed in Admin API v4
- The parsing code from `custom-redirects.js` has been moved to `frontend/services/redirects/settings.js`. This location is more appropriate for this logic and eventually `custom-redirects.js` middlewear might be moved into "frontend" as this middlewear plays a role mostly effecting that area.
no refs
[Portal](https://github.com/TryGhost/Portal) is a new drop-in script to make the bulk of Ghost membership features work on any theme out of the box, which was under a developer flag so far. This release removes the flag for Portal and makes it included as default for any members-enabled Ghost site. The Portal script is backward compatible with old public members script and existing Members-enabled themes should notice no change.
- Removes Portal config flag as Portal is now enabled by default
- Removes old members script as Portal is backward compatible with it
- Changes `{{content}}` helper to show default CTA in case of restricted content access
- `accent_color` setting is no more behind the dev experiment flag and included by default
- Adds migration to switch off Portal button setting for all existing sites which don't have Portal enabled in beta
closes#12263
- Express parses repeated query parameters as an array (req.query properties). Because there is no clear reason on why not to support this behavior extended order parameter parsing logic to handle arrays. This follows the rule of "liberal inputs, conservative outputs"
- Example supported query string for ordering can now look like: `?order=featured&order=published_at asc`, the priority of the order stays the same with the most significant appearing first and least significant last
refs #11572
- Filtering by fields coming from posts_meta table did not work for post resources. This was due to lack of support for these types of operations on NQL layer. The approach taken here is using same way filtering was done for many:many relations and generates a `WHERE IN` filtering clause. In the future we could look into adding preloading of 1:1 relations which should allow getting rid of `WHERE IN` in favor of `JOIN` and filtering directly by field names.
- Changed structure of `EXPANSIONS` filter configuration. Current approach was based on "bag of all the things". Such structure will become problematic as more fields are added. For example, adding all the fields from 1:1 relation posts:posts_meta might collide with any other relations that would have similar naming like meta_description from tags table (if it were was added).
- Bumped nql version to 0.5.0. This adds filtering support to 1:1 relations
- Added filter expansions which can be unique per model Previous approach with single global expansions lookup wasn't working in case different models would need to declare expansion for same field names. Having a `filterExpansion` method per model works in a similar convention other filter related model methods do (e.g. enforcedFilters, defaultFilters)
no issue
- adding user variables via the mailgun API when sending emails means that events related to email have those variables attached to them
- adding the `email.id` value to user variables means we can easily associate mailgun events to emails, otherwise we'd have look up the batch via `email_batches.provider_id` then use a join to get back to the associated email
- currently if Ghost has a version of something like
`3.37.0-pre.abc.def`, Ghost will return `3.37.0-pre.abc` as the full
version
- this hides parts of the version which are useful for debugging
- this commit updates the logic to join together all prerelease elements
so we keep the full string
no issue
The email change verification template was using the same as for `subscribe`, which did not have the right messaging that needs to be communicated about the action thats happening in this case. Updates the email template to same as what we use for email verification for support/newsletter address
closes https://github.com/TryGhost/Ghost/issues/12253
- Allows using custom action param for requests from Portal by using a new `requestSrc` option that is passed down when a request for magic link is made via Portal
refs https://github.com/TryGhost/Ghost/issues/12256 , https://github.com/TryGhost/Ghost/issues/12255
Currently when listing subscriptions for Members in both the Admin and the Theme, we only show the subscriptions which have a status of trialing or active.
Based on discussion, the `unpaid` and `past_due` states on Stripe also represent owner's intention of considering a subscription as active instead of `cancelled`, so we allow any subscriptions under these 2 states to be also listed for a member and consider them as `paid`.
- Subscriptions will go into a past_due state if the payment is missed, this should be considered a grace period where the member still has access.
- After this the subscriptions will either go to the unpaid or the cancelled state - this can be configured on an account by account basis in the Stripe dashboard. `unpaid` is considered as an intention to keep the subscription to allow for re-activation later.
no refs.
- Members settings were moved to a new page [here](https://github.com/TryGhost/Ghost-Admin/pull/1736)
- updated members related redirect for from/newsletter email address change to point to new page
no refs
- The token generation logic for single use token was replacing only the first instance of + or / to make the token URL safe, instead of replacing all instances which caused a bug where token was not validated properly in case it included multiple + or / in it.
- The fix ensures replacing all the + or / in the token with URL safe _ or - so it can be properly validated via magic link
closes https://github.com/TryGhost/Ghost/issues/11548
- bumps @tryghost/kg-default-cards
- introduces a breaking change where `imageOptimization` is passed as an options argument instead of separating out individual config options
- fixes width/height attributes for images in gallery cards being larger than the max default width used when resizing images
no issue
- centralises definition of max width and allows customisation if needed
- allows for passing of the config value through to rendering libraries
no issue
- standard browse/read/add/edit/destroy API endpoints for snippets resource
- updates `@tryghost/admin-api-schema` dependency to version that includes snippet definition and schemas
no issue
- all staff users can browse/read snippets so they're usable in the editor for everyone
- only administrators, editors, and admin integrations are able to create/edit/delete snippets
no issue
- minimal table structure required for the first iteration of content snippets
- snippets are stored pieces of re-usable content that could effectively be entire posts so the `mobiledoc` field length matches the `posts.mobiledoc` field length
no issue
- Member email settings - newsletter and support - was previously being set to null for all cases, which can lead to `null` values being set for those addresses on a successful import whereas the expected behavior is to retain the old values.
- Adds the 2 email settings to ignored settings list so they are not changed during import process
no issue
- In a case where support address is null, member's site data won't load as there was no fallback.
- The fix adds default `noreply` as the support address for site data
no-issue
The Action model loops through all registered models when the file is
loaded, by loading the model last, we ensure that it can read all
models, rather than an arbitrary selection which come before it.
closes#12273
- `comped` field has been allowed when editing a member or importing from a CSV. There has been a usecase (Zapier Integration) for API client to create a member with "Complimentary" plan, which made this change necessary
- Previously the logic for comped field was to skip and continue member record creation if Stripe was not connected. Now we throw an error - same as the one we have been throwing before when stripe_customer_id field was passed in. The implication of this change is that we won't be creating any record now if comped === true and Stripe is disabled.
- Bumped admin-api-schema-package. Contains `comped` schema change so this field gets passed through to controller
no issue
When scheduling a post to publish+send the "view online" link was pointing at https://site.com/404/ rather than the published post's url.
The problem occurred because the `/schedules/` endpoint wraps it's post read+edit calls in a transaction. Context:
- when a post is published with with the "send email" option the email record is immediately generated and added to the API response, as part of the email record generation we render the email content including fetching the url for the "view online" link
- urls for all resources are handled by our `url` service, that service updates it's internal cache based upon model events such as the "edited" event triggered when a post is published
- if the posts API controller is given a transaction, the email record is also generated inside of that transaction however at this point the `url` service will not have been updated because the post record hasn't been committed meaning it has no available url for the post
Fix:
- removed the `models.Base.transaction()` wrapper around the post read+update in the `/schedules/` API controllers
- we don't need a transaction here. It was added as protection against another write request coming in between the `/schedules/` controller reading a post and publishing a post but we already have protection against that in the form of collision detection - if a write request comes in and commits between the schedules controller reading the post and updating it, the scheduler's update call will fail with a collision error at which point the scheduler itself should retry the request which could then publish the post successfully if everything else is in order
closes https://github.com/TryGhost/Ghost/issues/12260
- if a card type was not explicitly chosen (i.e. a url was pasted into the editor) then abort fetching the oembed endpoint if we detect it's a `wp-json` oembed and return a bookmark card payload instead
- cleaned up an unused argument in the internal `fetchBookmarkData()` method
closes https://github.com/TryGhost/Ghost/issues/12257
- there was a destructuring problem introduced in the recent email refactor which meant the correct replacement data was not being passed over to the Mailgun provider when sending email
closes https://github.com/TryGhost/Ghost/issues/12259
- adds a `DISTINCT` to the query used to fetch member rows when generating an email recipient list
- this increases query time 2.7s vs 1.6s locally with ~94k paid members but once the `members.paid` column is implemented this slow query can be removed
closes https://github.com/TryGhost/Ghost/issues/12247
- Internal preview controller was lacking "mapping" call to post object which handled not only missing meta attribute information but lots of other mappings (e.g. users, tags, etc.)
- Have added a regression test to catch issues like this in the future
no issue
- fixed passing of errors up through send/processBatch/processEmail
- fixed errant overwrite of email status with a "submitted" status after a failure had occurred
no issue
- wrap email batch/recipient record creation in a transaction so if an error occurs during creation we're not left with a partially created batch/recipient set in the database
no issue
- if an error occurred whilst creating email batch/recipient records the email status was never updated and was left in the 'pending' status
- adjusted the error handling to update the email status and record the error message if such a scenario occurs
no issue
- the paid-member SQL query that is obtained using `models.Member.getFilteredCollectionQuery({paid: true})` can return multiple columns with the same name (eg, `email`, `name`), when that happens the last column with duplicate names "wins" and it's value is used in the resulting knex row instance
- in the `mega` service when fetching email recipient rows we ran into this problem, to avoid it we adjust the query to explicitly select only the data from the `members` table
no issue
- we've had an issue with emails failing due to unexpectedly missing data when inserting email recipient rows
- added a validation check before adding recipient details along with a log so that invalid data can be investigated
no issue
- We had previously allowed accent_color setting for member site settings behind portal flag, but Ghost Admin also needs the public site setting with accent color to correctly reflect the accent color when flag is switched on
- Removes deletion of accent color setting when behind the Portal flag OR dev experiment flag
no issue
- there was an typo in the recent email sending refactor that resulted in `Error: 'to' parameter is missing` errors when sending email previews and bulk emails
refs #12249
This was incorrectly assuming the presence of the data-members-name
element in the document. By guarding against it and defaulting to
undefined, we fallback to the existing behaviour when the element is not
present.
closes#12244
As per RFC 6454 the Origin header MUST be set to the string 'null' when
in a "privacy-sensitive" context. We were not handling this string and
this was causing errors. This commit updates all checks of the 'Origin'
header to treat the value 'null' as if the header was not present.
ref: https://tools.ietf.org/html/rfc6454#section-7.3
refs #12055
As part of the work in TryGhost/Members#206 we load the stripeCustomers relation on the member model, and we do not want this to be part of the API response. The changes here include a refactor but the main thing is that the serialized object is explicit and does not include unexpected or unknown fields.
* Moved mapMember out of mapper file
This cleans up the serializer a bit by keeping it's functionality all in
one place, rather than a shared mapper file
* Refactored members controller to return models
Previously the controller was calling toJSON, which is serialization,
this updates the controller to only deal with models, leaving all of the
serialization to the serializer!
* Refactored members serializer
This adds typings to all of the methods/functions in the serializer, as
well as making the serializating explicit, rather than returning the
result of toJSON, we explicitly set the properties we expect to be on
the output object. This protects us against accidental API changes in
the future.
closes https://github.com/TryGhost/Ghost/issues/10628
- JSON Schemas were extracted into a separate module to allow other clients to reuse them (for example documentation). Having them in a separate package also slims down the amount of code needed to be maintained in the core.
- Updated canary API input validators to use admin-api-schema module
- Removed canary schemas that moved into admin-api-schema package
- Updated v2 API input validators to use admin-api-schema package
- Removed v2 schemas that moved into admin-api-schema package
- Updated tests to contain needed information in apiConfig to pick up correct validation
- Added @tryghost/admin-api-schema package dependency
refs https://github.com/TryGhost/Ghost/pull/12232
When viewing sent emails in Ghost's admin area, it displays the `html` field directly from the `email` relation loaded with the post. Since the `mega` refactor we now store raw content in that field rather than sanitized "preview" content so it's necessary to modify the API output to match the old behaviour.
- use the API output serializers to parse replacements in email content and replace with the desired fallback or empty string
no issue
- store raw content in email record
- keep any replacement strings in the html/plaintext content so that it can be used when sending email rather than needing to re-serialize the post content which may have changed
- split post email serializer into separate serialization and replacement parsing functions
- serialization now returns any email content that is derived from the post content (subject/html/plaintext) rather than post content plus replacements
- `parseReplacements` has been split out so that it can be run against email content rather than a post, this allows mega and the email preview service to work with the stored email content
- move mailgun-specific functionality into the mailgun provider
- previously mailgun-specific behaviour was spread across the post email serializer, mega, and bulk-email service
- the per-batch `send` functionality was moved from the `bulk-email` service to the mailgun provider and updated to take email content, recipient info, and replacement info so that all mailgun-specific headers and replacement formatting can be handled in one place
- exposes the `BATCH_SIZE` constant because batch sizes are limited to what the provider allows
- `bulk-email` service split into three methods
- `send` responsible for taking email content and recipients, parsing replacement info from the email content and using that to collate a recipient data object, and finally coordinating the send via the mailgun provider. Usable directly for use-cases such as test emails
- `processEmail` takes an email ID, loads it and coordinates sending related batches concurrently
- `processEmailBatch` takes an email_batch ID, loads it along with associated email_recipient records and passes the data through to the `send` function, updating the batch status as it's processed
- `processEmail` and `processEmailBatch` take IDs rather than objects ready for future use by job-queues, it's best to keep job parameters as minimal as possible
- refactored `mega` service
- modified `getEmailData` to collate email content (from/reply-to/subject/html/plaintext) rather than being responsible for dealing with replacements and mailgun-specific replacement formats
- used for generating email content before storing in the email table, and when sending test emails
- from/reply-to calculation moved out of the post-email-serializer into mega and extracted into separate functions used by `getEmailData`
- `sendTestEmail` updated to generate `EmailRecipient`-like objects for each email address so that appropriate data can be supplied to the updated `bulk-email.send` method
- `sendEmailJob` updated to create `email_batches` and associated `email_recipients` records then hand over processing to the `bulk-email` service
- member row fetching extracted into a separate function and used by `createEmailBatches`
- moved updating of email status from `mega` to the `bulk-email` service, keeps concept of Successful/FailedBatch internal to the `bulk-email` service
requires https://github.com/TryGhost/Ghost/pull/12192
- added initial `EmailBatch` and `EmailRecipient` model definitions with defaults and relationships
- added missing `post` relationship function to email model
- fetch member list without bookshelf
- bookshelf can add around 3x overhead when fetching the members list for an email
- we don't need full members at this point, only having the data is fine
- if we need full models later on we can push the model hydration into background jobs where recipient batches are fetched ready for an email to be sent
- bookshelf model instantiation of many models blocks the event loop, using knex directly keeps concurrent requests fast
- adds `getFilteredCollectionQuery` method to base model to facilitate getting a knex query based on our normal model filters along with transaction/forUpdate applied
- store recipient list before sending email
- chunk already-fetched members list into batches and insert records into the `email_recipients` table via knex
- chunked into batches of 1000 to match the number of emails that Mailgun accepts in a single API request but this may not be the absolute fastest batch size for recipient insertion:
| Batch size | Batch time | Total time |
| ---------- | ---------- | ---------- |
| 500 | 20ms | 4142ms |
| 1000 | 50ms | 4651ms |
| 5000 | 170ms | 3540ms |
| 10000 | 370ms | 3684ms |
- create an email_batch record before inserting recipient rows so we can effeciently fetch recipients by batch and store the overall batch status
no issue
- The accent_color setting was being removed from members site data when behind portal flag
- The accent color setting is now allowed in members site data for all cases as it doesn't make any sense to remove it specifically from here where we already have all the other Portal settings included which is a dev/portal flag feature anyways
no-issue
By using the "email" validation, we were validating emails in CSV
imports using a different validator to the rest of the API. AJV's built
in email validation was failing on emails with "special" characters,
such as letters with an umlaut above them.
This commit brings the validation for CSV imports in line with the rest
of the API.
no issue
The email table should be a reference for all data that was used when sending an email. From and Reply-to addresses can change over time and we don't have any other reference for their value at the time of sending an email so we should store them alongside the email content.
- schema updated with `from` and `reply_to` columns
- both are set to `nullable` because we don't have historic data (can be populated and changed in later migrations if needed)
- neither `from` or `reply_to` have `isEmail` validations because they can have name+email in an email-specific format
- will help keep concerns separated in the future. `mega` service can deal with all of the email contents/properties, and the `bulk-email` service's concerns are then only email sending and any provider-specific needs
refs #12033
- Allowing to change parent integration opens up possible security holes and has no clear usecase at the moment. After a webhook record is created it should not be possible to change parent integration.
- Had do partially duplicate JSON schema definition from webhooks definition as there is no proper composition technique available in current version of JSON Schema.
refs https://github.com/TryGhost/Ghost/issues/12033
refs https://github.com/TryGhost/Ghost/issues/10567
- Creating a webhook without valid parent integration leads to orphaned webhook records, which shoult not ever happen
- This scenario is only possible for non-integration authentication,
because in case of integration being authenticated it's id is
automatically assigned to creatd webhook
refs #11729
- When ordering is done by fields from a relation (like post's `meta_title` that comes form `posts_meta` table), Bookshelf does not include those relations in the original query which caused errors. To support this usecase added a mechanism to detect fields from a relation and load those relations into query.
- Extended ordering to include table name in ordered field name. The information about the table name is needed to avoid using `tableName` within pagination plugin and gives path to having other than original table ordering fields (e.g. order by posts_meta table fields)
- Added test case to check ordering on posts_meta fields
- Added support for "eager loading" relations. Allows to extend query builder object with joins to related tables,
which could be used in ordering (possibly in filtering later). Bookshelf does not support ordering/filtering by proprieties coming from relations, that's why this kind of plugin and query expansion is needed
- Added note about lack of support for child relations with same property names.
closes#12038
Previously we were emitting changed events for _all_ settings which would
cause any listeners for those to be triggered, this ensures that listeners are
only triggered if the corresponding setting, _did_ in fact change.
no-issue
This change allows themes to collect member names on signup with use of the
`data-members-name` attribute on an input element in the `data-members-form`
refs #11729
- Having a plugin allows to reason about ordering better and gives way
to further extraction away form the core.
- Just like with filtering, ordering falls into similar category of having effect on knex'es query builder object extension through additional statements - ORDER BY in this case.
- Because there were bugs associated with use of permittedAttributes inside of `parseOrderOption` method, introduced separate `orderAttributes` function which returns only those fields which are orderable (https://github.com/TryGhost/Ghost/issues/11729#issuecomment-685740836)
refs #11878
- When password reset link is invalid previous messaging left the user
without clear information about why the reset failed and what they could do about it.
- Updated messaging around password reset tokens including detection of
when password token has invalid structure, has expired or has already
been used
This reverts commit 80af56b530.
- reverting temporarily so that all associated functionality can be merged in a single release
- creating email batch/recipient records without using them would cause inconsistent data
no-issue
* Added SingleUseTokenProvider to members service
This implements the TokenProvider interface required by members-api to
generate magic links. It handles checking if the token is expired and
pulls out any associated data.
Future improvments may include the email in the error for expired
tokens, which would make resending a token simpler.
* Passed SingleUseTokenProvider to members-api
This sets up the members-api module to use the new single use tokens
* Installed @tryghost/members-api@0.30.0
This includes the change to allow us to pass a token provider to the members-api
no-issue
This is a model for the tokens table, which handles the single use
aspect by customising the `findOne` method to automatically destroy the
model after reading from it
no-issue
refs: https://github.com/TryGhost/Members/commit/63942f03
The above commit updated all magic-link methods to be async, this change
ensures that we will handle a Promise return when it's updated in Ghost.
`await` has no effect on non Promise return values so it's safe to add
this now.
no-issue
After discussion with Matt, we decided that 192 bits for the token is a
good number, as it has no padding when base64 encoded and is more secure
than 128 bits, whilst still a managable size.
no-issue
This is a table to store single use tokens for use in magic links, the
columns are as simple as possible at the moment and are designed as:
id - standard ObjectID like all of our tables
token - 128bit base64 encoded string
data - arbitrary data to store against the token
created_at - timestamp to allow for expiry to be implemented for tokens
no issue
- In a recent change to ownership verification email flow, we changed the FROM address of ownership verification mails to use the same email as the one we are verifying, aka TO address.
- Email clients like Gmail flags off such emails as possible spam
- Fix updates the `FROM` address to `noreply@domain.com` where domain.com is domain for TO address
- In case the TO is already noreply@domain.com, we use no-reply@domain.com to bypass the same address restriction.
no issue
- The new Portal config flag allows switching on Portal conditionally with config
- The dev experiment flag still works for enabling Portal
- The flag currently defaults to `false` as Portal is still a beta feature and switched off by default
- We expose it on the admin api config endpoint so that the Ghost-Admin client can use it to conditionally render Portal settings
no issue
- Members site data was not appending blog domain for default support address which is `noreply`
- The change allows Portal to use default support address correctly
requires https://github.com/TryGhost/Ghost/pull/12192
- added initial `EmailBatch` and `EmailRecipient` model definitions with defaults and relationships
- added missing `post` relationship function to email model
- fetch member list without bookshelf
- bookshelf can add around 3x overhead when fetching the members list for an email
- we don't need full members at this point, only having the data is fine
- if we need full models later on we can push the model hydration into background jobs where recipient batches are fetched ready for an email to be sent
- bookshelf model instantiation of many models blocks the event loop, using knex directly keeps concurrent requests fast
- store recipient list before sending email
- chunk already-fetched members list into batches and insert records into the `email_recipients` table via knex
- chunked into batches of 1000 to match the number of emails that Mailgun accepts in a single API request but this may not be the absolute fastest batch size for recipient insertion:
| Batch size | Batch time | Total time |
| ---------- | ---------- | ---------- |
| 500 | 20ms | 4142ms |
| 1000 | 50ms | 4651ms |
| 5000 | 170ms | 3540ms |
| 10000 | 370ms | 3684ms |
- create an email_batch record before inserting recipient rows so we can effeciently fetch recipients by batch and store the overall batch status
no issue
We want to store a list of recipients for each bulk email so that we have a consistent set of data that background processing/sending jobs can work from without worrying about moving large data sets around or member data changing mid-send.
- `email_batches` table acts as a join table with status for email<->email_recipient
- stores a provider-specific ID that we get back when submitting a batch for sending to the bulk email provider
- `status` allows for batch-specific status updates and picking up where we left off when submitting batches if needed
- explicitly tying a list of email recipients to a batch allows for partial retries
- `email_recipients` table acts as a join table for email<->member
- `member_id` does not have a foreign key constraint because members can be deleted but does have an index so that we can efficiently query which emails a member has received
- stores static copies of the member info present at the time of sending an email for consistency in background jobs and auditing/historical data
refs #2635
- Adds 'Location' header to endpoints which create new resources and have corresponding `GET` endpoint as speced in JSON API - https://jsonapi.org/format/#crud-creating-responses-201. Specifically:
/posts/
/pages/
/integrations/
/tags/
/members/
/labels/
/notifications/
/invites/
- Adding the header should allow for better resource discoverability and improved logging readability
- Added `url` property to the frame constructor. Data in `url` should give enough information to later build up the `Location` header URL for created resource.
- Added Location header to headers handler. The Location value is built up from a combination of request URL and the id that is present in the response for the resource. The header is automatically added to requests coming to `add` controller methods which return `id` property in the frame result
- Excluded Webhooks API as there is no "GET" endpoint available to fetch the resource
closes#12045
- When member's email is updated to an already existing email of different member it caused table's unique constraint error, which was not handled properly.
- Added handling for this error similar to one in members `add` method.
closes#11999
- When the routes.yaml file changes (manually or through API) we need
to store a checksum to be able to optimize routes reloads in the future
- Added mechanism to detect differences between stored and current routes.yaml hash value
- Added routes.yaml sync on server boot
- Added routes.yaml handling in controllers
- Added routes hash synchronization method in core settings. It lives in core settings
as it needs access to model layer. To avoid coupling with the frontend settings it accepts
a function which has to resolve to a routes hash
- Added note about settings validation side-effect. It mutates input!
- Added async check for currently loaded routes hash
- Extended frontend settings loader with async loader. The default behavior of the loader is
to load settings syncronously for reasons spelled in 0ac19dcf84
To avoid blocking the eventloop added async loading method
- Refactored frontend setting loader for reusability of settings file path
- Added integrity check test for routes.yaml file
closes#12060
- A 500 error what happening when invited user provided an email that is associated with an existing user
- Additional validation for existing email address was added to prevent invalid data hitting db constraint error
refs #11999
- The `routes_hash` setting will be used during the boot process to update the hash
of currently loaded routes.yaml file in case it's different from last restart
no issue
- The async/await syntax makes it easier to reason about the code. Because adding 'Location' header is in the works it's a prep-work in a sense
closes https://github.com/TryGhost/members.js/issues/94
- The member-api package was recently updated to work directly with models and needs explicit `withRelated` options to attack relations
- Without options, the endpoint was returning the default member data without subscriptions attached, which in Portal showed paid member as free
- Fix updates the middleware for updating member data to correctly pass the relations needed to populate the member
no issue
- By default for new sites, support address is set same as from address to `noreply` , with full email address using the domain for `@`
- For newsletter emails, the support address was missing the default site domain to be added to address if its `noreply`
- Fix updates the support address to use the same format as from address and add relevant domain for default case
no refs
- The `update` method in members-api package was edited to return Model object instead of JSON directly - TryGhost/Members@a28bcc5
- This unsubscribe handler was returning the raw member object returned from `update` method, which is now a model object and not able to access `member.email`
- Fix updates the unsubscribe request handler to return the member JSON again
no issue
- members who have trial subscriptions added directly via Stripe will have a status of `"trialed"` in their Ghost subscription
- the `paid: true` filter was not taking that into account meaning trial users were not receiving newsletters sent to paid members even though they have a "paid" subscription
no issue
- We used existing "from" address as sender for mails sent to new email address for verification, but that breaks the flow to update if the current "from" address has DMARC policy set.
- This updates the flow to always send the ownership verification email TO the new address and FROM the new address which both verifies the email deliverability for new address and ownership
no issue
- The newsletter emails are sent out with `from` address as sender
- The new `members_reply_address` setting is now used to set reply-to address for emails, which can be either newsletter or support address
no issue
- Member auth emails were previously using the `from` address as sender
- New `members_support_address` was introduced with default as original "from" address
- Auth emails use the new support address as sender
no issue
- Updated magic link generation and validation methods for email update API to handle new support address
- Updated importer to ignore the new support address as it can only be updated via verification
- Updated members service to listen on settings edit for new support/reply address fields as well
- Updated tests to include the new settings
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account, public setting
- `members_reply_address` - Where you receive responses to newsletters
closes#12167
- Tags API v2 was ignoring `count.posts` include parameter.
- Regression was introduced with a3f693b472
- Introduced regression tests across all Content API versions to avoid similar bug in the future
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account
- `members_reply_address` - Where you receive responses to newsletters
no issue
- When an import was done and there were no "global labels" present Ghost created generic `import-[data]` label which later helped to find a specific batch of imported data
- It did not make sense to create such generic label when user provided their own unique label
- The rules that work now are:
1. When there is no global provided Ghost generates on and removes it in case there are no imported records
2. When there is a unique new global label provided no new label is generated, but the label stays even if there are no imported records
no issue
- tested performance between knex raw, knex `count()` and bookshelf `count()` and found no difference over 1000 iterations of each (each ~19,500ms +- 500ms for 104k members locally)
- switched to using bookshelf as the code is the simplest
no issue
- extract filtering of an collection into a separate function
- use extracted function in `findAll()` so that it's query behaviour matches `findPage()`
no issue
- for large result sets or complex queries the count query itself can be quite time consuming
- when `limit: 'all'` is passed as an option there's no need to perform a separate count query because we can determine the pagination data from the final result set
- skipped count query when `limit: 'all'` option is present
- re-ordered comments to be closer to the code they reference (ie, why we have our own count query instead of Bookshelf's `.count()`
no issue
- This is handled on input sanitization layer with date
format check in JSON schema validation, so there's no need to do this
check again in the importer.
no issue
- When bulk insert fails there is no transactional logic to revert
related records form being inserted. Also, previously there were no
attempts to "retry" the insert.
- To avoid complex retry logic, an iterative one-by-one insert retry
approach was taken. If this becomes a bottleneck in the future, the
retry algorithm could be improved.
- To avoid a lot of code duplication refactored model's `bulkAdd` & `bulkDestroy`
methods to use 'bulk-operations' module.
- Updated error handling and logging for bulk delete operations. It's very
unlikely for error to happen here, but still need to make sure there is
a proper logging in place to trace back the failure.
- Added debug logs. This should improve debugging experience and
performance measurements.
- Added handling for unrecognized errors. Handling inspired by current unrecognized
error handling by ghost importer -10e5d5f3d4/core/server/data/importer/importers/data/base.js (L148-L154)
no issue
- When no members are succesfully imported through CSV import process the import label should not be created. Otherwise after multiple failed attempts to import there are orphaned labels in the system
no issue
- updates `@tryghost/kg-default-cards` which contains two fixes
- removes email-specific output being added to post html (had no visual impact due to use of conditional comments but keeps rendered html smaller+cleaner)
- adds a background-url style to the thumbnail container to give two options for styling
- updates member email template styling to hide the `<img>` element in bookmark cards and use a background image instead to get consistent rendering across email clients
no issue
- adds a `members:emailTemplate` config object
- `showSiteHeader` - defaults to `true`, shows the site title and icon in member emails
- `showPoweredBy` - defaults to `false`, adds a "Publish with Ghost" button to member email footer
- updates member newsletter email template with hideable site header and "powered by" badge
closes https://github.com/TryGhost/members.js/issues/87
- The `update` method in members-api package was edited to return Model object instead of JSON directly [here](a28bcc5b2a)
- This caused the update member API on member endpoint to return partial response only as most properties couldn't be fetched
- Fix updates the middleware to correctly call `toJSON` before formatting response
* 3.30.2:
Bumped @tryghost/members-api to 0.28.1 in lockfile
Bumbed @tryghost/members-api to 0.28.1
🐛 Fixed unable to delete member when stripe is connected
refs #12127
- Adds new `editSubscription` endpoint for members admin API which allows updating individual subscription for a member - `PUT /members/:id/subscriptions/:subscription_id/`
- `editSubscription` has same permissions as member's `edit` endpoint
- Currently allows toggling of cancellation at period end for an active subscription
no issue
- Similar handling to one introduced in 8418c829de
- Having granular tracking for failed to remove id's would make it possible to return more specific errors to the client
no issue
- When batch insert fails handling should be more granular and aim to retry and insert as many records from the batch as possible.
- Added retry logic for failed member's batch inserts. It's a sequential insert for each record in the batch. This implementation was chosen to keep it as simple as possible
- Added filtering of "toCreate" records when member fails to insert. We should not try inserting related members_labels/members_stripe_customers/members_stripe_customer_subscriptions records because they would definitely fail insertion without associated member record
no issue
- When stripe is disconnected and there are Stripe-connected records present in imported set they should not be processed and proper error should be thrown
closes https://github.com/TryGhost/Ghost/issues/12139
- once the email content has been rendered in the post serializer, perform some whole-content transformation of `figure` and `figcaption` to `div` using cheerio
- juiced will have already inlined the elements styles so there's no need to adjust the template's stylesheet
closes#12078
- Root cause was that pseudo class .kg-bookmark-author:after was not getting inlined to email newsletter or its preview.
- That is where the margin and bullet-point content are added between author and publisher.
- Dependency juice supports an option inlinePseudoElements which is false by default.
- Fix was to set inlinePseudoElements to true when serializing post email.
Co-authored-by: Jeremy Davidson <jeremy@crossingcontour.com>
- testmode allows us to test Ghost's behaviour with long running requests and jobs
- moved all the testmode logging to only happen if Ghost starts successfully to make it clearer what is happening
- added a (currently very dirty) bit of code to fetch the jobService and output the queue status giving us a similar
output for both open connections and jobs
- all of this allows for debugging exactly what Ghost is doing whilst it is processing a shutdown request
closes https://github.com/TryGhost/Ghost/issues/11536
- bumps `@tryghost/kg-default-cards`
- image and gallery cards now output `width/height` attributes on `img` elements with a max width of 600px
- uses resized images where possible to keep email weight down
- adds `height: auto` style to image card images so that the `height` attribute does not cause distortion at smaller screen widths
no-issue
* Added bulkAdd method to Member,Customer&Subscription model
This allows us to keep the db access in the model layer
* Updated @tryghost/members-api to 0.27.2
This includes fixes for rate-limiting of requests, and exposes necessary
Stripe methods for creating customers and complimentary subscriptions,
without affecting the database.
* Refactored importer to parallelise tasks where possible
By parallelising our tasks we are able to improve the speed at which the
entire import completes.
refs #12126
- Adds migration to add impersonation permission to administrators
- Adds default permission fixture to allow administrators to read member impersonation urls
- Allows administrators to create member impersonation magic links
closes https://github.com/TryGhost/Ghost/issues/12129
Migrations were failing when upgrading from a version that didn't include stripe member subscriptions table. The `up` migration was checking for the existence of a unique index that was added to the schema in 3.29 but it was using the wrong variable name which meant that it would never return true for an existing index. For most migrations this worked because the index existed but when through 2.34 the table was created from scratch and did included the index at that point.
- fixed variable name and re-ordered variable assignment for better code locality that would have made the typo more visible
no issue
- Additional validation is needed for imported data because in case of bulk insertions (through knex) we bypass model layer validation - this could lead to invalid data in the database, which would be hard to fix.
- Chose validation method we use for other endpoints - through JSON Schema. It proved to be very performant (200ms overhead for 50k records). When comparing it with iterative method (validating each record separately) this was adding about 17s of overhead.
- Refactored returned values from "sanitizeInput" method to encapsulate more logic so that the caller doesn't have to calculate amount of invalid records and deal with error types
- Whole sanitizeInput method could now be easily extracted into separate module (somewhere close to members importer)
- Bumped members-csv package. It is meant to handle empty string values - '' and null, which should allow validating member records more consistently!
- upon further testing, this started to not work again so I'm removing
this horrible hack
- it could be Bunyan playing around with us, and the server is actually
shutting down
- I have no idea why this is required and I really don't like it
- this change looks superfluous but I had problems with Ghost not
displaying shutting down messages properly and this is the only
thing that got it working
- the async-awaits were getting stuck in `await this._cleanup()`,
and then wouldn't enter into the `finally` block
- oddly, this was only reproducible when a job was running, it would
shut down normally otherwise
- if I find the solution in the mean time, I'll happily get rid of this
no issue
- Updated sanitization logic to be self contained and return sanitieze input along with error stats
- This should give a nice place for validations to fit in
no issue
- There were many failed import records due to rate-limit errors. With concurrency of 9 imports go through with 100% success
- Would need to verify these limits with live API to make the most of it
no issue
- we output the post excerpt in a hidden div in the email template so that email clients pick it up as the "preview" text when listing emails
- when no custom excerpt is provided the preview text is grabbed from post.excerpt which is the first 500 chars of the post.plaintext value
- post.plaintext formats links as "Link [http://url/]" which is unwanted in html email previews
- add a basic replacement to the post email serializer to remove any `[http://url/]` occurrences from the post excerpt before rendering the email content
refs https://github.com/TryGhost/Ghost/issues/11536
- Outlook supports `'` as a special char for apostrophes but not `&#apos;` which is what cheerio/juiced render
- adds a basic string placement to the email serializer to switch to the older style of special char
no issue
- moves the meat of `pendingEmailHandler()` code into a new function `sendEmailJob()` that is passed over to the new job service
- lets the server keep processing email generation and sending when it receives a shutdown request rather than halting processing mid-send and ending up in a partial state
no-issue
* Added stripeSubscriptions relation to member model
This allows us to fetch the subscriptions for a member via standard
model usage, e.g. `withRelated: ['stripeSubscriptions']` rather than
offloading to loops and `decorateWithSubscriptions` functions, this is
more performant and less non-standard than the existing method.
* Updated serialize methods to match existing format
The current usage of `decorateWithSubscriptions` and the usage of
members throughout the codebase has a subscriptions array on a stripe
object on the member, this ensures that when we serialize members to
JSON that we are using the same format.
There is definitely room to change this in future, but this is an
attempt to create as few breaking changes as possible.
* Installed @tryghost/members-api@0.26.0
This includes the required API changes so that everywhere can use
members-api directly rather than models and/or helper methods
no issue
- The code in controller was becoming hard to reason about.
- Having a single module shows exactly how many dependencies are there to do an import for single batch.
- Having a separate module would make it easier to extract into it's own package in Members monorepo
no issue
- Member's labels have to have sort_order assigned when added/edited. This was lacking from batched importer.
- Implementation is based on logic used in model's base - e484709e73/core/server/models/base/index.js (L81-L86)
- Bottom line - we need to manage shutting down gracefully when doing long-running tasks
- To achieve that, we're going to use job queues
In this commit:
- added new @tryghost/job-manager dependency
- added a minimal job service, that handles in passing things like logging and (maybe later) config
- job service is wired up to server shutdown, so that the queue finishes before the server exits
- also added a new job endpoint to testmode so that it's easy to test job behaviour without needing to do real work
- Using consistent patterns for shutdown and stop
- Make it clear what each method does
- Use async/await to make the code more readable and simple
- This lays groundwork for having more cleanup tasks in stop than just server.stop()
- deleted files under `core/server/lib/promise` and related test files
- added `@tryghost/promise` as a dependency
- fixed all local requires to point to the new package
closes#12118
- server.start was mistakenly removed in 71f02d25e9
- it is used for loading themes (and other things) and is critical
- added tests to prevent this regressing again in future
no issue
- New Member API batched import is meant to be a substitution to current import
with improved performance while keeping same behaviore. Current
import processes 1 record at a time using internal API calls and times
out consistently when large number of members has to be imported (~10k
records without Stripe).
- New import's aim is to improve performance and process >50K
records without timing out both with and without Stripe connected
members
- Batched import can be conceptually devided into 3 stages which have
their own ways to improve performance:
1. labels - can be at current performance as number of
labels is usually small, but could also be improved through batching
2. member records + member<->labels relations - these could
be performed as batched inserts into the database
3. Stripe connections - most challanging bottleneck to solve because
API request are slow by it's nature and have to deal with rate limits of
Stripe's API itself
- It's a heavy WIP, with lots of known pitfalls which are marked with
TODOs. Will be solved iteratively through time untill the method can be
declared stable
- The new batched import method will be hidden behind 'enableDeveloperExperiments' flag to
allow early testing
- A simple way to test behaviours without having to do complex interactions to e.g. generate errors or slow requests
- Makes it easier to test the new shutdown behaviour, among other things
- there can now be quite a big delay between SIGINT/TERM being received and shutdown finishing
- add an extra log message to acknowledge the SIGINT/TERM to facilitate debugging and just be clear
- stopppable is a dependency that handles closing connections properly, which server.close does not
- active connections are allowed to complete what they are doing
- idle connections are closed
- no new connections are allowed
- we call stoppable in stop() instead of server.close so that idle connections don't hold the server open
- calling await stop() from shutdown then ensures that we have a consistent experience of stop
- all together this allows ghost to shutdown gracefully when there are long-running requests
- @TODO: handle graceful shutdown of long-running processes
- @TODO: consider do we need to send 503s whilst the server is shutting down?
- changed method to logStopMessages, as we use start and stop, not start and shutdown
- changed logStopMesasges to output the "proper" messages and use this method consistently - the closing connections message isn't really useful
- changed uptime message to always be output cos I can't see a case where there isn't interesting/useful
- Connection handling is legacy code added in 1438278ce4
- Although we were tracking all connections in memory, we weren't actually closing any because stop isn't called
- This (and restart) were both added as part of the now long-deprecated system for using Ghost directly as an npm module
- If we want to close connections cleanly, we should use a tool to do this
- the announce functions exist for the purpose of communicating with Ghost CLI
- the functions were called announceServerStart and announceServerStopped, which implies they tell Ghost CLI when the server starts and stops
- however, the true intention / purpose of these functions is to:
- either tell Ghost CLI when Ghost has successfully booted (e.g. is ready to serve requests)
- or tell Ghost CLI when the server failed to boot, and report the error so that Ghost CLI can communicate it to the user
- therefore, I've refactored the old functions into 1 function to make it clearer they do the same job, but with 2 different states
- also added some tests :D
refs 022a433e56
- noticed I missed adding debug info in one place (not that it's used, just for consistency)
- also made the SIGINT/TERM code slightly more readable
no-issue
- switch from `membersService.api.members.list` to using bookshelf `Member.findPage()` with the `{paid: true}` filter to avoid per-member queries (N+1) to decorate members with subscriptions and a heavy post-fetch filter via `contentGating`
- add concurrency to the Mailgun API requests in `bulk-email` service to reduce overall time submitting API requests
- add debug statements with timing output for easier measurements
no issue
For large numbers of members we're unable to perform queries for free/paid members in a reasonable time without indexes. Bulk delete is also very slow when looping through bookshelf models and having bookshelf manage deletion of child records, adding `ON DELETE CASCADE` to the foreign key indexes moves child deletion to the DB level and allows for performant bulk delete queries.
- migrations only run on mysql because sqlite does not support altering tables
- adds foreign key indexes and constraints with 'ON DELETE CASCADE' for members_labels, members_stripe_customers, and members_stripe_customers_subscriptions
refs #12100
For performance reasons we want to add foreign key and unique constraints
to the members_stripe_* tables so we can utilised cascading deletes and
joins across the tables when querying.
In order to do this we must first ensure that:
- There are no duplicate entries in the `subscription_id` or `customer_id` columns
- There are no orphaned rows in the subscription or customers tables
If the first is not true, the unique constraint will fail, and if the second is not true,
the foreign key constraint will fail.
As we are only adding the indexes to existing MySQL databases at this point, the
cleanup migrations will also only be done for existing MySQL databases too.
The migrations for removing orphaned rows splits the deletion into a `SELECT`
followed by a `WHERE IN` to avoid the database "optimising" the query into a
`JOIN` which ends up taking much longer due to the lack of indexes.
no-issue
We are in the process of creating migrations to add foreign key constraints
and cascading deletes to the members_stripe_* tables to make listing members
and deleting members faster. As well as the migrations we need to update the
database schema so that new installations have the correct indexes and constraints.
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
refs 173e3292fa
- The bug was initially introduced in referenced commit. When request is done with `api_key` context, there should always be an `integration` object associated with it - 71c17539d8/core/server/services/permissions/parse-context.js (L36) . An `id` from `context.integration` not `context.api_key` has to be assigned to newly created webhook!
- The webhooks API is about to be declared stable in upcoming release, so no migration will be done
closes#11907
The image in the bookmark card was being shown out of the bounds of
the card because of a general style `height: auto !important`.
I added a new `max-height` property to the image to avoid exceeding
parent height.
no issue
- Webhooks API has been stabilized with latest changes and there are no breaking changes planned for v3. The change has strictly "informative" purpose
- Changed variable naming from "whitelisted" to "allowlisted" to follow updated naming convention (refs. https://mysqlhighavailability.com/mysql-terminology-updates/)
refs #12064
- `critical` is meant to be something unpredictable like internal error, something worthy attention, as described in Ignition -3439456d94/README.md (list-of-errors)
- This error level was introduced with - this PR https://github.com/TryGhost/Ghost/pull/9426, but there is no context provided why this specific value was used. Assuming it's an outdated value as 'not found' is nowhere to be treated in any special way
no issue
- `mailgun()` expects the `host` option not to include a port but `url.host` will include the port, we instead want to use `url.hostname` which skips the port
no issue
- bookshelf adds `DISTINCT` to any relation query that does not have an explicit `columns` statement
- when measuring the impact of `DISTINCT` on the eager-loading association query when listing members using `{withRelated: 'labels'}`, it can be 2x slower with no index on the sort_order column or 4x slower with an index on sort_order
no issue
- When processing entries with new labels in parallel Bookshelf relations is trying to create them which caused unique key constraints to fail. To avoid the failure, all labels should be pre-created before proceeding with creating members
- Member limit code was duplicated in 2 places unnecessarily
- Also used member api code that fetched members and subscriptions fully hyrated when we only need a count
- Using a raw query significantly improves performance here
no-issue
This updates the Admin API Member resource to *not* cancel subscriptions
by default, and adds a `cancel` option. This can be used over HTTP by
including a `cancel=true` query parameter.
- mailgun has a testmode flag we can use to get email to be accepted but not delivered
- this is useful for developers testing general bulk email code - not for users - so it is only available via config
no-issue
Up until now we have left orphaned rows in members_stripe_* tables when
a member is deleted, this updates the destroy method so that we cascade
and remove any MemberStripeCustomer and StripeCustomerSubscription
models related to the Member.
This also adds regression tests for the new functionality as well as to
confirm the existing functionality of cascading to the members_labels
join table
This adds the relations of Subscription->Customer & Customer->Member
refs https://github.com/TryGhost/members.js/issues/72
- Portal is using using publication logo from settings for signup/signin pages
- Instead, we are switching to using publication icon from settings, which also needs to be passed in site data API
no issue
Having all members created during an import labelled with a specific "import label" is useful for later operations such as bulk delete/edit or simply recording how and when a member was created.
- automatically create a label with the date/time the members CSV import occurred and assign it to all imported members
- return the import label data in the API response so that clients can react accordingly such as automatically filtering the members list by the label once an import finishes
refs #12074
Since we've split members settings into multiple keys the
reconfiguration of the members-api has been happening in quick
succession as the stripe_connect_* settings are all set at once.
This debounces the call to reconfigure the members-api so that we only
need to instantiate it once.
no issue
- By default, GhostMailer throws EmailError with statusCode as `500` for any failure in sending mail
- In case of failure due to `RecipientError`, status code as now correctly sent as `400` as its a bad request and not an error we can't handle.
closes#12049
Stripe plans used to default to 0, and our new validation of plan
amounts were causing issues when importing from an older version of
Ghost, this updates the validation to be skipped when importing.
- Added regression test for importing plans
refs https://github.com/TryGhost/Ghost/issues/11971
- Added statusCode from bulk email provider to API response
- Updated error messages for different bulk email(mailgun) failure states
- Added `context` to preview mail API error message with mail provider's error message
closes#12033
- Added webhooks schemas and definitions.
- Added validation checking if integration_id is present when using session auth. This is needed to prevent orphan webhooks.
- Integrated webhook schemas into frame's validation layer.
- Added isLowerCase ajv keyword support. This is needed to be able to do isLowerCase validation using JSON Schema for webhooks.
closes#12001
* Moved settings validation to the model
This moves the settings validation out of the validation file and into
the model, as it is _only_ used there.
It also sets us up in the future for custom validators on individual
settings.
* Improved validation of stripe_plans setting
- Checks `interval` is a valid string
- Checks `name` & `currency` are strings
* Moved stripe key validation into model
The stripe key settings are all nullable and the regex validation fails
when the input is `null`. Rather than reworking the entirety of how we
validate with default-settings validation objects, this moves the
validation into methods on the Settings model.
* Added tests for new setting validations
Adds tests for both valid and invalid settings, as well as helpers
making future tests easier and less repetitive
closes#12015
refs 95880dddeb
- The bug was caused by falsy plaintext field assignment to empty string `''` when the html content was `null`. Because of the `setEmptyValuesToNull` function (referenced commit), there is no sense to assign empty string value to plaintext property, because it would still end up being `null`
- The `''` -> `null` conversion was confusing the model layer to think that some fields were changed, where in reality none did. This in turn lead to a bug with falsy cache invalidation
closes#12016
- The change detection didn't work when editing post_meta fileds because we only check current model's `_changed` fields when performing `wasChanged()` check
- A solution was adding change tracking of post_meta relation to currently edited post model and overloading `wasChanged` method to check these fields as well
refs #12043
- On updating From-address from Members settings in Labs, we send a confirmation email to the updated address with magic link for verification
- Previously, no explicit sender email was being set for this so fallback config address was used
- This updates the sender address to use the current from address for "from-address" update emails
no-issue
- Added breaking test for webhook url including subdirectory
- Previously the webhook handler URL was generated incorrectly when
running Ghost on a subdirectory, appending the path to the root of the
host, this fix ensures that the subdirectory is included before the
path.
no-issue
This version of members-api includes changes to how webhooks are
managed, previously they would be deleted and recreated on every boot of
Ghost. Now they are created and the secret is persisted, on boot the
webhook is updated to the most current url and events. If the api
version is wrong or the update fails, the webhook is deleted and
recreated and the settings updated.
- Installed @tryghost/members-api@0.24.0
- Updated config to work with 0.24.0
no-issue
They will be used to store webhook information so that we can persist it between
boots and simplify the creation process of webhooks in members
closes https://github.com/TryGhost/Ghost/issues/12026
- 3 new portal settings were added in `3.23` in default settings - `portal_button_style`, `portal_button_icon`, `portal_button_signup_text`
- New settings default to group `core` on migrating from pre 3.22 versions due to missing columns in DB
- Migration here updates the new settings to correct group
closes#11994
- Adds support for ordering based on slug filter that contains a slug-is-in filter. It is applied only to Content API's resources - post, page, tag, author. The order is applied in the same order in which slugs appear in the filter.
- For, example providing following query parameter filter for any of the above resources: `?filter=slug:[kitchen-sink,bacon,chorizo]`, would filter them by these slugs and order in the same way defined in the filter
- Can be used in handlebars templates in following way: `{{#get "tags" filter="slug:[slugs,of,the,tags,in,order]"}}`
- The property conteining this new order is assigned to `autoOrder` instead of `rawOrder` intentionally. This explicit asstignment would allow distinguishing where the 'orderRaw' comes from the model or the API layer. Apart from adding necessary context this separation makes it easier to refactor separately model layer and API specific ordering in the future
- This commit also fixes default filtering for `author` resource in Content API. The serializer was never used before as it was missing from `serializers/index.js` module.
no issue
- Changes introduced to both API v3 and v2
- Makes sure to use the same integration_id as authenticated integration for the webhook's data.
- Makde it is impossible to create orphaned webhooks using token authentication
- Allowed only parent integration to edit it's children webhooks. Throwing permission error otherwise
no-issue
There is concern that the settings cache can return `null` for values
which it cannot parse correctly, this just ensures that we always have
an array where we expect one
no issue
- Existing and new sites should have portal button hidden by default and need explicit switch on from Admin.
- Default value for new sites has been updated, but existing sites still have old default and portal button set as true
- Migration here resets portal button setting to false for all existing sites, and they'll need to be explicitly turn it on from admin again
no issue
- Adds new portal button settings to members site data for portal script
- Updates settings input/output serializers to handle portal icon image url
no issue
- Adds new portal settings - `portal_button_style`, `portal_button_icon` and `portal_button_signup_text`
- New settings allows customization of portal button
- Updates tests to include new settings
* tag '3.22.2':
v3.22.2
Updated Ghost-Admin to v3.22.2
Emitted all settings events on reinit of cache (#12012)
🐛 Updated access to be true by default in v3 API
Hardened members subscription migration against missing data (#12009)
closes#12003
There are a few parts of Ghost that rely on the settings events being
emitted anytime a setting is changed, so that the data is kept in sync.
When a setting is renamed in a migration essentially what happens is
that the settings value is changed from a default value to its actual
value, but this does no emit an event.
Anything that is initialised before migrations have run that relies on
the events to keep it up to date will have stale data - e.g. the themes
i18n service.
This change ensures that when we `reinit` after migrations have been
run, we emit events for every setting to tell the rest of Ghost that it
has changed.
closes#11990
- access should be a members feature, but it was already accidentally exposed to the theme layer
- it has now been added to the API even if members is disabled
- access defaults to true, unless members is enabled
- when members is enabled, access is set to the currently logged in members' access
closes#11993
We had some issues with some databases being in an unexpected state this
check for each property before using it add uses defaults when it is
missing.
no issue
- when searching for paid/free members, the `members_stripe_customers`
table would be joined into the query on `members`
- this table also has a `name` and `email` field, so both MySQL and
SQLite would complain about ambiguous fields in the query
- the result of this would be a 500 error thrown inside Ghost, and no
useful response to the user
- this commit explicitly chooses the `members` table to check against,
and also adds a test for this
no issue
- The membersjs default trigger button was set to be visible by default, even behind dev flag
- This changes the default visibility to false so it needs to be switched on from Admin to be visible on screen
closes https://github.com/TryGhost/Ghost/issues/11944
- updates `@tryghost/image-transform` to version that exposes `canTransformFiles()` which checks for `sharp` availibility
- updates `@tryghost/kg-default-cards` to version that accepts a `canTransformImage()` method as an option
- updates our `mobiledoc` lib to pass a `canTransformImage()` function that returns false if sharp is unavailable, the image extension is not supported, or the storage engine in use does not support image transforms
- updates `populateImageSizes` to fetch image sizes when transforms are unavailable as the render/not-render is now handled in the renderer and we don't need to worry about adding size information to the mobiledoc source
refs https://github.com/TryGhost/Ghost/issues/10318
- re-initialize settings cache after migrations by shutting down to clean up event listeners then and calling `init` again
- important to ensure `db.ready` event is not emitted until settings have finished re-initializing to avoid problems with background processes using the db connection which is disconnected/re-connected or being kicked off with out-of-date settings
refs 05f6faf846
- The logic in "Stripe" error detection depended on error's message string matching, which is not a good practice in general. Had fixed it do to exact match on "context" of the error we throw internally and left more extensive comment about why things are implemented the way they are.
no issue
Output serializer's url util was expecting `og_image` and `twitter_image` to be top-level attributes in the `attrs` object but they are actually nested under `posts_meta`.
- updated the code to use lodash's `get/set()` so that we can work with paths for easier handling of nested objects
- fixed unit tests where the mocked data under test did not match real-world data
closes#11932
- as per the issue, there is no need to output a <link> tag if the favicon is the default /favicon.ico, as all browsers automatically check for this
- instead the favicon <link> is only output if a custom favicon has been set, telling the browser to look somewhere different to its default location
- some of the tests expected 3 links in the html head, but through the favicon change there are often only 2.
Co-authored-by: RenCloud <rencloud@pop-os.localdomain>
closes#11574
- the current implementation of the access property has it frontend only, and wired up only in one place
- this leaves it only available in a handful of places, e.g. can't use it in a post loop or get helper
- the current implementation also fails logically if the html content of the post is blank
This fix moves the behaviour to the API
- this ensures the field is always available no matter what context you are in
- it also updates the logic to use the same membersHasAccess logic as is used to gate the post, so it's always correct
TODO: should reconsider the location of this code
refs https://github.com/TryGhost/Ghost/issues/10318
- Skipped `members_subscription_settings` and `stripe_connect_integration` from settings import
- Fixed import data field filtering for core and theme groups
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates `boolean` serialization in v2/canary serializers to apply only for `boolean` type settings
- Updates `boolean` transformation in model layer `format`/`parse` to check on `boolean` type setting
- Removes error thrown on Read-only setting for settings edit endpoint
- Updates v2/canary input serializers to remove any Read-only settings (using RO flag) to avoid edits
- Added type/group mappings in the importer when pre-migration settings table import data is present
- Updates tests
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates default settings to contain correct type and validation for each setting
- Updates `populateDefaults` to correctly insert type value for new settings
- Updates settings schema to allow only select types - `array`, `number`, `boolean`, `string`
- `object` is a temporary type allowed till we get rid of all JSON object settings
refs #10318
Because settings are not populated with the correct group and flags, we
must _always_ set these. Then we can check to see if there are values
which need migrating, and if not, can safely exit and leave the values
as default.
refs #10318
If migrating from a previous version that does not include the setting
being migrated from we can safely not update the new setting, and just
rely on its default value being present. When rolling back we can use
defaults if the new setting does not exist.
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates the type field for a setting to specify its type explicitly
- Type can be one of (string, number, boolean, array)
- `object` is allowed a temp type till all the object settings are removed
- existing `type` value for setting was misleading and referring to `group`
- we moved the current `type` value for a setting to `group` here - 4a9e57c170
* Updated members default settings
ref #10318
This pulls out the members_subscription_settings & stripe_connect_intgration settings into separate keys
* Updated usage of members_from_address
* Updated stripe_connect usage
* Updated members config to use new settings
* Updated members middleware to use isStripeConnected
* Updated members service to reload correctly
We reload the members-api instance when the related settings change, so
this makes sure we're listening to the correct settings changes
* Updated ghost_head helper to use new settings
* Updated theme middleware to use new settings
* Renamed members_allow_signup -> members_allow_free_signup
* Fixed tests after settings refactor
* Removed from direct key settings key
* Fixed regression tests for settings api
refs #10318
As populateDefaults is run _before_ migrations, the new settings will
already be inserted in the database, so we just need to update their
values and then delete the old settings.
refs https://github.com/TryGhost/Ghost/issues/10318
`Settings.populateDefaults()` is run before migrations during Ghost's startup. This can cause problems when new settings table columns are added (and populated in `default-settings.json`) because `populateDefaults()` was using the model layer which assumes that those columns are available, resulting in `ER_BAD_FIELD_ERROR: Unknown column` type errors.
- query the database for the available `settings` table columns
- switch to using raw knex queries without Bookshelf for insertions so that we're in control of the columns that are added
- use `_.pick` to skip any properties in `default-settings.json` that do not match to an available column - those columns will be added and populated by later migrations
- moving away from using the model to insert settings has the side-effect of not emitting `settings.added/edited` and `settings.x.added/edited` events, this should be fine because `populateDefaults()` is called before anything else is set up and listening
- added a call to `populateDefaults()` in our knex-migrator "before migration" hook so that we have consistent db state across both startup initialised migrations and manually triggered knex migrations
no issue
- Import error grouping wasn't working correctly when error didn't have defined context property
- The copy will be refined in final design review
no-issue
These utils are the first steps toward getting the models out of our
migrations! The utils here interact directly with the database and where
possible don't reinvent the wheel, by using smaller building blocks to
build more comples ones.
refs TryGhost/Ghost#10318
refs 8fc526ff6
- This is symetric change to one done for v3 API (commited as 8fc526ff6)
- Added 'core' filtering for v2 API controller
refs https://github.com/TryGhost/Ghost/issues/10318
- `shared_views` is always an array (Ghost-Admin has handling to revert it to an array if it's anything else) but it's default value was set to an empty object
refs https://github.com/TryGhost/Ghost/issues/10318
- `members_public_key` and `members_private_key` are now correctly grouped under `core` which is already filtered out so we don't need the extra filter to exclude them
refs https://github.com/TryGhost/Ghost/issues/10318
refs 1dc0405803
- Adds 1:1 mapping for filtering options to renamed settings "type" to "gorup"
- Ignores the name changes and any old types
- Detailsed type -> group mappings can be checked in the refereneced migration commit
refs https://github.com/TryGhost/Ghost/issues/10318
- maps old `settings.type` values to new `settings.type/group` values
- uses an explicit map so that we don't lose information and can safely roll back even though we're modifying `settings.type` too
- updates `settings.type` values too to keep code working while we switch to using `settings.group`
- sets the `settings.group` value for all settings which are keeping the same group as their current type
- adapts `settings.type` validations to match new groups
- adds flags to specific settings, both in the migration for existing settings records and in default-settings.json for new settings records
refs https://github.com/TryGhost/Ghost/issues/10318
- `group`
- to replace the `type` column, provides a more descriptive name for the columns use
- for existing sites it will be populated by migrating data from the `type` column in a later migration
- for new sites a minimal update has been added to `parseDefaultSettings()` to populate the `group` field when settings are created during startup - fixes the NOT NULL constraint on `settings.group`
- `flags`
- signifies special handling that is different to other settings in a group
- eg, `PUBLIC,RO` would indicate that the setting is available via unauthenticated endpoints and is read-only
refs https://github.com/TryGhost/Ghost/issues/10318
- There was a copy/paste error and we didn't have a test to pick it up. Will follow up with a regression test to make sure it doesn't happen again
refs #10318
refs 2614565d5a
- Adds importer mapping for fields as in referenced migration
- The intention is to allow exports from v2/v3 to still be compatilbe with current version
- Adds a mapper for deprecated fields and imports them with new settings "keys"
refs #10318
refs 2614565d5a
- Renamed ghost_head/ghost_foot in settings to match the new names
introduced in migrations
- Above change lead to reshufling in the mappings in input/output
serializers
- Makes sure change is compatible with v2 API
refs https://github.com/TryGhost/Ghost/issues/10318
refs 2614565d5a
- Renames to match referenced migration renames
- Fixed API responses so they are consistent with newly renamed fields
- Not returning lang and timezone keys from settings in API v2 ther rest should be returned in API v3/canary
refs https://github.com/TryGhost/Ghost/issues/10318
- precursor to migrating from `settings.type` to `settings.group`
- renames `blog` type to `site`
- renames `bulk_email` type to `email`
- moves settings out of `site` (previously `blog`) into more appropriate groups such as `core` or individual feature groups
no issue
- Previous migration in commit - 2614565d5a - was using incorrect json structure for brand -> primary_color
- Fixes the up/down migrations to work on correct existing `brand: primary_color` structure
refs https://github.com/TryGhost/Ghost/issues/10318
- Adds following renames to settings table keys:
'default_locale' -> 'lang'
'active_timezone' -> 'timezone'
'ghost_head' -> 'codeinjection_head'
'ghost_foot' -> 'codeinjection_foot'
'brand.publicationColor' -> 'accent_color'
- The renames are done to match revised naming conventions and naming
exposed through APIs
- Supersedes this revert - 1eeb5a60b8 and #11946
no issue
- when using subdirectories, images can be stored in the database both with and without the subdirectory prefix. We weren't taking that into account and so images without the subdirectory were not having the `/content/images/` prefix removed when passed to the storage adapter resulting in the storage adapter not finding the image
refs https://github.com/TryGhost/Ghost/issues/10318
- Adds following renames to settings table keys:
'default_locale' -> 'lang'
'active_timezone' -> 'timezone'
'ghost_head' -> 'codeinjection_head'
'ghost_foot' -> 'codeinjection_foot'
'brand.publicationColor' -> 'accent_color'
- The renames are done to match revised naming conventions and naming
exposed through APIs
refs #10318
refs https://github.com/TryGhost/Ghost/pull/11942
- Removes force_i18n, permalinks, and members_session_secret usage from the codebase
- We deprecated these flasgs and have not used since Ghost v2. It's good time to remove them before we introduce bigger changes to how `settings` table opeartes.
- Fixed importer test. The test was meant to check if string values were converted properly, the check agains boolean didn't make much sense in this context, so removed it.
- Following this change are going to come ralated migrations to fix existing data (see ref)
no issue
- reverts commit 87c31444fd but with modifications to settings naming
- Adds new settings for members modal customization to default settings
- `portal_button` controls the visibility of beacon in members modal
- `portal_name` controls the visibility of name field in signup
- `portal_plans` controls the visibility of plans allowed for member to signup with
- Adds stripe connect check to determine if stripe is setup or not
- Adds the 3 new settings to members site data
- Updates to snake case naming for members site API data
no issue
- Moves out CSV parsing and serialization related code into separate package as a part of push to modularize Ghost repo.
- Next up is to remove `csv-parser` dependency from this new package
refs 7904c303a7
- Model's validation errors are sometimes returned as an array of errors, in those cases they should be flattened to before returning to the client. This way there is more insight into what went wrong during the import.
no-issue
pr: https://github.com/TryGhost/Ghost/pull/11930
- Upgraded @tryghost/members-api to 0.23.0
This version includes a new method hasActiveStripeSubscriptions
- Added /admin/members/hasActiveStripeSubscriptions
This can be used to determine whether or not we should allow removing
the stripe keys.
- Added /admin/settings/stripe/connect
This can be used to delete a Stripe Connect integration, provided
there are not active subscriptions
no issue
- fixed incorrect method name when calling
- fixed problem with setting post.mobiledoc to a promise rather than waiting for the size population to finish and setting it to a mobiledoc string
no issue
- `?formats` is useful when you want to get `html` back from the Admin API when creating/editing posts, otherwise you're forced to make a following `GET` request
no issue
- Adds new settings for members modal customization to default settings
- `membersjs_show_beacon` controls the visibility of beacon in members modal
- `membersjs_show_signup_name` controls the visibility of name field in signup
- `membersjs_allowed_plans` controls the visibility of plans allowed for member to signup with
- Adds stripe connect check to determine if stripe is setup or not
- Adds the 3 new settings to members site data
no issue
- adds `populateImageSizes()` to our mobiledoc lib module
- uses `image-size` lib to speed up reading of image dimensions
- for local images, use storage adapter with same guards as used by `handle-image-sizes` middleware so that we don't insert srcsets for images that aren't transformable
- for unsplash images, remove any width and crop params from the url so it points to the full-size image
- use `populateImageSizes(mobiledoc)` to modify post model's mobiledoc when re-rendering
no issue
- known failing case was Medium. Some *.medium.com articles triggered redirects but without cookies being stored/sent across the redirect requests it would cause an infinite redirect loop and we'd abort after 10 redirects
- use `got`'s [cookie support](https://github.com/sindresorhus/got/tree/v9.6.0#cookies) via `tough-cookie` so that we can create bookmark cards for medium.com and other sites with similar problems
- There were various cases where it was possible to trigger a private site to display a 404 instead of redirecting to /private/
- Private mode was also not always displaying the correct robots.txt
- This PR includes tests for all cases in test/frontend-acceptance/default_routes_spec.js & where possible the unit tests have also been updated for completeness
- Fixing the 404 issues required
- Better handling of paths using req.path instead of req.url in filterPrivateRoutes
- Additional error handling, to cover the case that a tag/author RSS feed does not exist
- Fixing the robots.txt required the order of middleware to be changed, so that private blogging gets a chance to render first
- NOTE private blogging is the only app with a setupMiddleware function so nothing else is affected
no-issue
This issue only occurs when using custom redirects with a subdirectory
setup, and the path to be redirected from is expressed as a regex, and
the url that is being redirected to is not an external url.
The issue has a few components:
- Redirect paths as a regex generally use the ^ to ensure that they
match the beginning of the path.
- The path that the regex is matched against conditionally excludes the
subdirectory, specifically, the subdirectory is excluded for external
urls
These combined means you end up with a regex like /^\/custom-redirect/
and a path like /subdir/custom-redirect, these will not match/replace
correctly, and you'll end in an infinite redirect loop.
The fix here is to *always* remove the subdirectory when testing regex's
and then conditionally adding it back *only* for the redirect, and only
if it is an internal redirect
no issue
- localhost check was causing an immediate "no provider" error when fetching oembed for a localhost url and bypassing the bookmark fallback even when the configured site lives on localhost
- allow `localhost:port` through in oembed endpoints when it matches the configured url
no issue
- pass `srcsets` value through to mobiledoc renderer
- it will stop adding `srcset` attributes to images when explicitly set to `false`
- adds `reload()` method to `mobiledocLib` so that memoized instances of the renderer can be reset during tests that change config values
closes#11917
- Pass text-only version to mailgun as `text` not `plaintext`
- This ensures we send a text-only version of the email, and this in turn should help to improve spam scores
refs https://github.com/TryGhost/Ghost/issues/11414
- Importing data currently overwrites the existing "from address" with new value
- "from address" needs to go through email validation flow before update which was bypassed
- Updates importer to not allow overwrite for "from address" and use existing
- Adds test for "from address" overwrite
- We have many customers asking for INR as there are special rules in Stripe for this currency
- As well as a desire for local-selling
- Meaning it's not valid to use e.g. USD instead
no-issue
This was initially missed as local settings always had the original
stripeDirect keys, this ensures that regardless of Connect vs Direct vs
Both vs Neither that the config is correct.
Also ensures that the Members API instance is reloaded when the Stripe
Connect settings are changed.
no issue
- there are various situations where we adapt/fix/improve our mobiledoc->html output over time but we didn't have a way of updating old content without manually editing the mobiledoc and saving, or running an expensive migration to re-render all old content
- this adds a `?force_rerender=true` query param to the `PUT .../admin/posts/:id/` endpoint that allows the `html` field to be re-generated without modifying the `mobiledoc` field contents
no issue
- The intention is to move away from using file type names in URLs. This endpoint is meant to correspond to current `POST /members/csv` endpoint, that is planned to be renamed into `/members/upload`. And the `GET /members/csv` to be renamed to `/members/download` respectively.
no issue
- NQL does not support the relationship setup that members->stripe customer<->stripe subscriptions uses so it wasn't possible to use the `filter` param to query against having an active subscription
- adds `customQuery` bookshelf plugin that allows customisation of SQL query used in `findPage` method by individual models
- use `customQuery` in Member model to set up joins and conditionals to select free/paid members when `options.paid` is present
- allow `?paid` param through API and permitted options for member model
no issue
- This new format allows to return additional metadata with failed import records. The data for invalid records is returned in following format:
```
{
count: {count_of_invalid_records},
errors: [{
message: "Members not imported. Members with duplicate Stripe customer ids are not allowed." // message field of the error
context: "Attempting to import members with duplicate Stripe customer ids." // context field of the error
help: "Remove duplicate Stripe customer ids from the import file, and re-run the import." // help field of the error
count: 2 // count of this specific error
}]
};
- Errors are grouped by their context fields because message fields sometimes can contain unique information like Stripe customer id, which would produce too many errors in case of bigger datasets.
no issue
- This endpoint is meant to be used for validation of imported members
- Main function at the moment is to validate if stripe_customer_id present in the dataset exists in connected Stripe account
no issue
- Ghost-Admin redirects all paths to `/ghost/settings...` as `/ghost/#/settings/...`, this updates the admin redirect on successful magic link validation to directly use the latter to avoid extra redirect
no issue
- adds a set of hardcoded "content image sizes" to the base config
- adjusts `handle-image-sizes` middleware to always allow the hardcoded content image sizes to be genreated
- updates `@tryghost/kg-card-factory` to allow passthrough of options to card renderers
- updates `@tryghost/kg-default-cards` to add `srcset` output for image and gallery cards
closes#11008
- Updated @nexes/nql to 0.4.0
This version exports the mapKeyValues utility function
- Replaced nql-map-key-values with @nexes/nql util fn
Usage was found using `rg nql-map-key-values` and replaced globally.
- Deleted nql-map-key-values module in shared
Now that this module isn't referenced anywhere else, we can remove it,
relying solely on the util exported by @nexes/nql
refs https://github.com/TryGhost/members.js/issues/43
- Adds new `isStripeConfigured` flag to public members site data which denotes if stripe setup is completed
- Helps clients like members.js/themes to configure payment behavior based on this flag
no-issue
The service at stripe.ghost.org must know which client_secret to use,
either the test, or live one. By encoding a JSON object as the state we
are able to pass data through the flow to inform this decision at the
end.
Note, that we still keep a random value in the state to protect against
CSRF attacks.
no-issue
We've added a "mode" query param to the members_stripe_connect api auth
method, allowing the client to easily switch between live and test mode.
no-issue
This adds the ability to pass a `mode` param of 'test' to the members stripe
connect service, which will ensure we use a testmode client_id for the
Stripe Connect OAuth flow.
This will allow users to connect their account in testmode.
no issue
- When imported member contains stripe_customer_id data but there is no Stripe configured on the Ghost instance such import should faiil. The logic is consistent with one where import fails after not being able to find customer in linked Stripe account
- Fixed import stats to show import failures instead of "duplicate" when the validation error is of "Stripe" origin
no issue
- The aim was to extract format-csv to become an external dependency. After some analysis found out that native papaparse method `unparse` was achieving the same results with a lot better test coverage and stability. Because papaparse will become Ghost's default csv processor in near future decided to integrate papaparse instead of extracting module for the format-csv module's code, which would become redundant soon anyways.
- For reference papaparse will substitute current csv-parser lib because it's better performance and maturity.
- Performance comparison can be checked here - https://github.com/Keyang/csvbench#result . At the time of writing papaparse is rougly 40% faster than csv-parser
refs https://github.com/TryGhost/Ghost/pull/11807
- this was originally reported in the PR above, but we could not
reproduce it on master
- presumably the user had the latest version of moment installed for
other purposes and so they were seeing the issue
- between moment 2.24.0 and 2.26.0, something must have changed which
stopped the previous functionality working
refs https://github.com/TryGhost/Ghost/issues/11414
Confirms if the fromAddress for sending member emails is valid and accessible using magic link flow, allowing owners to update full from address including domain change.
- Extends member service to handle magic link generation and validation for email update
- Updates existing setting endpoint to not directly update from address
- Adds new endpoint to send magic link to new address
- Adds new endpoint for validating the magic link when clicked and update the new email for from address
- Adds new email template for from address update email
refs https://github.com/TryGhost/Ghost/issues/11414
We want to allow adding custom domains to member's from address as long as the new email is verified using the magic link flow and not saved directly to DB. Since we currently don't store domain as part of fromAddress, this PR -
- adds migration to update all existing fromAddress by appending site domain
closes https://github.com/TryGhost/Ghost/issues/11768
- Wraps from parameter in double quotes so mail clients can read it whole
- Escapes double quotes in site title to avoid clash with wrapping
refs https://github.com/TryGhost/Ghost/issues/11212
- if a bookmark card fetch is performed (either directly or from fallback) and the page does not have an extractable title, return a more specific error message than "No provider found for supplied URL."
no issue
- When the customer cannot be imported because they are missing from linked Stripe account or the linked account is incorrect one, these new messages should provide a better clue about what has caused the error and how to act on it.
no issue
- There is a need to be able to label certain import group of members
with custom labels. This will allow to distinguish/filter these newly
imported members.
- Allowed `POST /members/csv/` endpoint to accept `labels`
field parameter which assigns labels to every member from imported csv.
no issue
- Similarly to other additive api methods (e.g. members.add) returned more specific ValidationError with contex filled in with the reason why adding did not succed.
- This change is needed for more graceful label handling when adding new members through import
no issue
- Adding labels doesn't cause any content to invalidate, similarly to adding members. Unlike it's caunterpart - tags, there is no dependent "frontend" content that would become invalid
no-issue
This was because the API was recieving a default `null` value for the
stripe_connect_integration_token setting.
This also improves the logging for this error.
no issue
- added an `externalRequest` lib
- uses same underlying `got` module as our `request` lib
- uses `got`'s `beforeRequest` and `beforeRedirect` hooks to perform it's own dns resolution for each url that's encountered and aborts with an error if it resolves to a private IP address block
- includes a bypass for Ghost's configured url so that requests to it's own hostname+port are not blocked
- updated v2 and canary oembed controllers to use the `externalRequest` lib
no-issue
If the stripeDirect config value is NOT set / false, then connect token
should be used over the API keys if both are present, or else use
whichever is present. If the stripeDirect config value is set /
true,then API keys should be used and stripe connect token should be
completely ignored.
The reason for this is that we want to use Stripe Connect as the primary
auth method going forward, but we want to keep the direct version
availiable should the Ghost Foundation ever dismantle. This will allow
people to use all the same features with no dependency on an external
service.
no-issue
In order to issue a redirect we need access to the "raw" req/res
objects, which is why we must return the function which gets access to
them.
The members service is used to create the auth url and to update the
users session.
no-issue
Uses the members service to parse a stripe_connect_integration_token
setting and set the stripe_connect_integration based on that.
This change includes ignoring the stripe_connect_integration{,_token}
settings, as the token is never saved, and the integration can only be
set by using the token.
no-issue
This module handles the creation of a url used for authorization of
Stripe Connect, and also the parsing of the data eventually received
from the authorization flow.
no-issue
There was some unused code here, the variable was never used, also we
were looping and collecting a list of errors, but only every using the
first one, so switched to the `find` method which stops iteration after
an element has matched.
closes https://github.com/TryGhost/Ghost/issues/11841
- Migration adds mapping between permissions and roles for email_preview send test mail
- Only owner previously had correct permission to send test emails
- Fixture existed to allow Admin/Editor/Integrations to send test mails but had missing migration
- Adds tests for roles to send test email
refs https://github.com/TryGhost/members.js/issues/30
- Added new `updateEmail` type for sending email address update confirmation mail to member
- The link in email updates member's email address
no issue
- node was complaining the package was missing after the sanitize-html update [1]
- the update removed an old version of lodash which supported the dot importing
method
- our code relied on this subdependency to work, but the structure has since
been updated
[1]: https://github.com/TryGhost/Ghost/pull/11867
- Represents that logging is shared across all parts of Ghost at present
* moved core/server/lib/common/logging to core/shared/logging
* updated logging path for generic imports
* updated migration and schema imports of logging
* updated tests and index logging import
* 🔥 removed logging from common module
* fixed tests
refs https://github.com/TryGhost/members.js/issues/30
- Member cannot update their email directly but need to do it via magic link sent to new email address
- Previous profile update change had allowed email to be updated directly as well for authenticated member
no issue
- adds `search` bookshelf plugin that calls out to an optional `searchQuery()` method on individual models to apply model-specific SQL conditions to queries
- updated the base model's `findPage()` method to use the search plugin within `findPage` calls
- added a `searchQuery` method to the `member` model that performs a basic `LIKE %query%` for both `name` and `email` columns
- allowed the `?search=` parameter to pass through in the `options` object for member browse requests
* moved `server/config` to `shared/config`
* updated config import paths in server to use shared
* updated config import paths in frontend to use shared
* updated config import paths in test to use shared
* updated config import paths in root to use shared
* trigger regression tests
* of course the rebase broke tests
* refactored core/frontend/services/proxy to import common dependency like a normal person
* removed all imports of `common/errors`
* 🔥 removed common/errors module
Co-authored-by: Vikas Potluri <vikaspotluri123.github@gmail.com>
no issue
- moves members stats generation for the admin graph from the client to the server
- outputs a basic totals count across a requested date range of 30, 90, 365 days, or all time. See below for the response shape
- leaves heavy lifting of the counts to the SQL engines - tested on a dataset of 100k members and query performance is <100ms
```
GET /ghost/api/canary/members/stats/?days=30
{
total: 100000,
total_in_range: 20000,
total_on_date: {
'2020-04-25': 19000,
'2020-04-26': 19500,
// continues until today's date
},
new_today: 200
}
```
* refactored `core/frontend/apps` to destructure common imports
* refactored `core/frontend/services/{apps, redirects, routing}` to destructure common imports
* refactored `core/frontend/services/settings` to destructure common imports
* refactored remaining `core/frontend/services` to destructure common imports
* refactored `core/server/adapters` to destructure common imports
* refactored `core/server/data/{db, exporter, schema, validation}` to destructure common imports
* refactored `core/server/data/importer` to destructure common imports
* refactored `core/server/models/{base, plugins, relations}` to destructure common imports
* refactored remaining `core/server/models` to destructure common imports
* refactored `core/server/api/canary/utils/serializers/output` to destructure common imports
* refactored remaining `core/server/api/canary/utils` to destructure common imports
* refactored remaining `core/server/api/canary` to destructure common imports
* refactored `core/server/api/shared` to destructure common imports
* refactored `core/server/api/v2/utils` to destructure common imports
* refactored remaining `core/server/api/v2` to destructure common imports
* refactored `core/frontend/meta` to destructure common imports
* fixed some tests referencing `common.errors` instead of `@tryghost/errors`
- Not all of them need to be updated; only updating the ones that are
causing failures
* fixed errors import being shadowed by local scope
no issue
- Removes global bodyParser middleware for membersApp and adds it to specific endpoints
- Removes global boolParser middleware for membersApp
We added bodayParser middleware to memebrsApp in [this](fe3eab1836) commit to read json requests for members update endpoint, but that had issues with stripe webhook parsing for `/webhooks` endpoint as stripe expects raw data to be passed down.
no issue
- updated `filter` plugin with appropriate label relationship and replacement config for NQL
- fleshed out member and label fixture data and tools to facilitate tests
no-issue
The flag currently defaults to `true` as we are still using stripe
direct. We expose it on the admin api config endpoint so that the
Ghost-Admin client can use it to conditionally render
- Allows member logged in with valid session to update their profile info - name, email, subscribed(newsletter subscription status)
- Adds new util method for formatted member response on the endpoints
- Adds common middlewares for body/bool parser and maintenance
- Adds `subscribed` status to member response
no issue
- shows the full URL instead of a truncated link in password reset email body. This is required for security and usability reasons (copy/paste)
closes#11825
- The initial implementation had a typo in a role name which didn't allow "Administrator" to edit post's "visibility" attribute
- Added unit tests to check administrator specific role and visibility attribute permission
- Add a query param that indicates whether signin/up succeeded or failed
- Add unit tests for all 3 possible cases for the createSessionFromMagicLink middleware
- Added an acceptance test to show the behaviour works in principle
- This restores the functionality from 3.14 as follows:
/members/ -> (with no route) rendered 404 error
/members/ -> (with route) renders members template
/members/?token=invalidtoken&foo=bar -> redirects to /?foo=bar
/members/?token=validtoken&foo=bar -> redirects to /?foo=bar
refs https://github.com/TryGhost/Ghost/pull/11790
- reduced complexity by sticking to one email for both normal reset and forced reset (locked staff accounts)
- exposed `siteTitle` for use in any email templates
- updated email copy to be suitable for both types of password reset
- Fixed session invalidation for "locked" user
- Currently Ghost API was returning 404 for users having status set to "locked". This lead the user to be stuck in Ghost-Admin with "Rousource Not Found" error message.
- By returning 401 for non-"active" users it allows for the Ghost-Admin to redirect the user to "signin" screen where they would be instructed to reset their password
- Fixed error message returned by session API
- Instead of returning generic 'access' denied message when error happens during `User.check` we want to return more specific error thrown inside of the method, e.g.: 'accountLocked' or 'accountSuspended'
- Fixed messaging for 'accountLocked' i18n, which not corresponds to the
actual UI available to the end user
- Added automatic password reset email to locked users on sign-in
- uses alternative email for required password reset so it's clear that this is a security related reset and not a user-requested reset
- Backported the auto sending of required password reset email to v2 sign-in route
- used by 3rd party clients where the email is necessary for users to know why login is failing
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
no issue
- Adds new endpoint on integration to refresh admin/content api key secret
- Allows owner/admin to refresh their content or admin API keys for an integration via Ghost Admin
- Adds a new `refreshed` event to actions table for anytime an api_key secret is refreshed
- Added a wrapper around express.Router to our shared/express util
- Also export static and _express
- Use this shared util everywhre, meaning express is only used directly in this one file
- ATM this file is mostly an experiment / debug helper, it might be removed again later
- The aim is to have a minimal framework wrapping express that allows us to:
- reduce our usage of express() in favour of Router()
- unify some of our duplicated logic
- fix some structural issues e.g. Sentry
- make it easier to understand the codebase
refs https://github.com/TryGhost/Ghost/issues/11756
- we removed the fixed `width: 600px` to fix gmail scaling but that means we need a new way of creating a 600px centre column in Outlook
refs https://github.com/TryGhost/Ghost/issues/11756
- fixed gmail scaling problems
- the `width: 600px` on `.container` was forcing gmail to always render at 600px wide and then use scaling to resize the email to fit the device width
- for most emails gmail would also apply their own font resizing to compensate so it didn't look _too_ bad
- some emails however would not trigger the font resizing, most notably when posts contained a feature image, which would result in very small text
- removing the fixed `width: 600px` resolves the scaling problem and lets the email be truly responsive
- removed attribute selectors in the media query CSS
- gmail does not support attribute selectors
- attribute selectors used to be necessary for Yahoo Mail but this is no longer the case
- tested using litmus.com for all popular email clients
- Use array destructuring
- Use @tryghost/errors
- Part of the big move towards decoupling, this gives visibility on what's being used where
- Biting off manageable chunks / fixing bits of code I'm refactoring for other reasons
- Meant to cleanup the old api/canary/members earlier, removed now as it's unused
- Also removed all the duplicate references to labs.members in various places
- My codebase-wide replacement of const/let yesterday went one step too far :)
- grunt uglify:prod fails on the use of const
- This file should continue to use var for the time being cos there's plans to upgrade this script already :)
- Clarify that the parent app has 2 distinct parts: backend and frontend
- Frontend app takes members and site apps + the frontend SSL redirect middleware
- Backend app already has admin + API (and the SSL redirect needs significant work)
- There's a lot more to do here, but this increases clarity
