no-issue
This column will allow us to decouple the recipients of newsletter from
the `visibility` of a post, allowing us to send emails to specifically
free members, or to send paid posts as newsletters to all members.
no refs
[Portal](https://github.com/TryGhost/Portal) is a new drop-in script to make the bulk of Ghost membership features work on any theme out of the box, which was under a developer flag so far. This release removes the flag for Portal and makes it included as default for any members-enabled Ghost site. The Portal script is backward compatible with old public members script and existing Members-enabled themes should notice no change.
- Removes Portal config flag as Portal is now enabled by default
- Removes old members script as Portal is backward compatible with it
- Changes `{{content}}` helper to show default CTA in case of restricted content access
- `accent_color` setting is no more behind the dev experiment flag and included by default
- Adds migration to switch off Portal button setting for all existing sites which don't have Portal enabled in beta
no issue
- all staff users can browse/read snippets so they're usable in the editor for everyone
- only administrators, editors, and admin integrations are able to create/edit/delete snippets
no issue
- minimal table structure required for the first iteration of content snippets
- snippets are stored pieces of re-usable content that could effectively be entire posts so the `mobiledoc` field length matches the `posts.mobiledoc` field length
no issue
- Member email settings - newsletter and support - was previously being set to null for all cases, which can lead to `null` values being set for those addresses on a successful import whereas the expected behavior is to retain the old values.
- Adds the 2 email settings to ignored settings list so they are not changed during import process
no issue
The email table should be a reference for all data that was used when sending an email. From and Reply-to addresses can change over time and we don't have any other reference for their value at the time of sending an email so we should store them alongside the email content.
- schema updated with `from` and `reply_to` columns
- both are set to `nullable` because we don't have historic data (can be populated and changed in later migrations if needed)
- neither `from` or `reply_to` have `isEmail` validations because they can have name+email in an email-specific format
- will help keep concerns separated in the future. `mega` service can deal with all of the email contents/properties, and the `bulk-email` service's concerns are then only email sending and any provider-specific needs
no-issue
After discussion with Matt, we decided that 192 bits for the token is a
good number, as it has no padding when base64 encoded and is more secure
than 128 bits, whilst still a managable size.
no-issue
This is a table to store single use tokens for use in magic links, the
columns are as simple as possible at the moment and are designed as:
id - standard ObjectID like all of our tables
token - 128bit base64 encoded string
data - arbitrary data to store against the token
created_at - timestamp to allow for expiry to be implemented for tokens
no issue
We want to store a list of recipients for each bulk email so that we have a consistent set of data that background processing/sending jobs can work from without worrying about moving large data sets around or member data changing mid-send.
- `email_batches` table acts as a join table with status for email<->email_recipient
- stores a provider-specific ID that we get back when submitting a batch for sending to the bulk email provider
- `status` allows for batch-specific status updates and picking up where we left off when submitting batches if needed
- explicitly tying a list of email recipients to a batch allows for partial retries
- `email_recipients` table acts as a join table for email<->member
- `member_id` does not have a foreign key constraint because members can be deleted but does have an index so that we can efficiently query which emails a member has received
- stores static copies of the member info present at the time of sending an email for consistency in background jobs and auditing/historical data
refs #11999
- The `routes_hash` setting will be used during the boot process to update the hash
of currently loaded routes.yaml file in case it's different from last restart
no issue
- Updated magic link generation and validation methods for email update API to handle new support address
- Updated importer to ignore the new support address as it can only be updated via verification
- Updated members service to listen on settings edit for new support/reply address fields as well
- Updated tests to include the new settings
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account, public setting
- `members_reply_address` - Where you receive responses to newsletters
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account
- `members_reply_address` - Where you receive responses to newsletters
refs #12126
- Adds migration to add impersonation permission to administrators
- Adds default permission fixture to allow administrators to read member impersonation urls
- Allows administrators to create member impersonation magic links
closes https://github.com/TryGhost/Ghost/issues/12129
Migrations were failing when upgrading from a version that didn't include stripe member subscriptions table. The `up` migration was checking for the existence of a unique index that was added to the schema in 3.29 but it was using the wrong variable name which meant that it would never return true for an existing index. For most migrations this worked because the index existed but when through 2.34 the table was created from scratch and did included the index at that point.
- fixed variable name and re-ordered variable assignment for better code locality that would have made the typo more visible
- deleted files under `core/server/lib/promise` and related test files
- added `@tryghost/promise` as a dependency
- fixed all local requires to point to the new package
no issue
For large numbers of members we're unable to perform queries for free/paid members in a reasonable time without indexes. Bulk delete is also very slow when looping through bookshelf models and having bookshelf manage deletion of child records, adding `ON DELETE CASCADE` to the foreign key indexes moves child deletion to the DB level and allows for performant bulk delete queries.
- migrations only run on mysql because sqlite does not support altering tables
- adds foreign key indexes and constraints with 'ON DELETE CASCADE' for members_labels, members_stripe_customers, and members_stripe_customers_subscriptions
refs #12100
For performance reasons we want to add foreign key and unique constraints
to the members_stripe_* tables so we can utilised cascading deletes and
joins across the tables when querying.
In order to do this we must first ensure that:
- There are no duplicate entries in the `subscription_id` or `customer_id` columns
- There are no orphaned rows in the subscription or customers tables
If the first is not true, the unique constraint will fail, and if the second is not true,
the foreign key constraint will fail.
As we are only adding the indexes to existing MySQL databases at this point, the
cleanup migrations will also only be done for existing MySQL databases too.
The migrations for removing orphaned rows splits the deletion into a `SELECT`
followed by a `WHERE IN` to avoid the database "optimising" the query into a
`JOIN` which ends up taking much longer due to the lack of indexes.
no-issue
We are in the process of creating migrations to add foreign key constraints
and cascading deletes to the members_stripe_* tables to make listing members
and deleting members faster. As well as the migrations we need to update the
database schema so that new installations have the correct indexes and constraints.
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
closes#12001
* Moved settings validation to the model
This moves the settings validation out of the validation file and into
the model, as it is _only_ used there.
It also sets us up in the future for custom validators on individual
settings.
* Improved validation of stripe_plans setting
- Checks `interval` is a valid string
- Checks `name` & `currency` are strings
* Moved stripe key validation into model
The stripe key settings are all nullable and the regex validation fails
when the input is `null`. Rather than reworking the entirety of how we
validate with default-settings validation objects, this moves the
validation into methods on the Settings model.
* Added tests for new setting validations
Adds tests for both valid and invalid settings, as well as helpers
making future tests easier and less repetitive
no-issue
They will be used to store webhook information so that we can persist it between
boots and simplify the creation process of webhooks in members
closes https://github.com/TryGhost/Ghost/issues/12026
- 3 new portal settings were added in `3.23` in default settings - `portal_button_style`, `portal_button_icon`, `portal_button_signup_text`
- New settings default to group `core` on migrating from pre 3.22 versions due to missing columns in DB
- Migration here updates the new settings to correct group
no issue
- Existing and new sites should have portal button hidden by default and need explicit switch on from Admin.
- Default value for new sites has been updated, but existing sites still have old default and portal button set as true
- Migration here resets portal button setting to false for all existing sites, and they'll need to be explicitly turn it on from admin again
no issue
- Adds new portal settings - `portal_button_style`, `portal_button_icon` and `portal_button_signup_text`
- New settings allows customization of portal button
- Updates tests to include new settings
* tag '3.22.2':
v3.22.2
Updated Ghost-Admin to v3.22.2
Emitted all settings events on reinit of cache (#12012)
🐛 Updated access to be true by default in v3 API
Hardened members subscription migration against missing data (#12009)
closes#11993
We had some issues with some databases being in an unexpected state this
check for each property before using it add uses defaults when it is
missing.
no issue
- The membersjs default trigger button was set to be visible by default, even behind dev flag
- This changes the default visibility to false so it needs to be switched on from Admin to be visible on screen
refs https://github.com/TryGhost/Ghost/issues/10318
- Skipped `members_subscription_settings` and `stripe_connect_integration` from settings import
- Fixed import data field filtering for core and theme groups
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates default settings to contain correct type and validation for each setting
- Updates `populateDefaults` to correctly insert type value for new settings
- Updates settings schema to allow only select types - `array`, `number`, `boolean`, `string`
- `object` is a temporary type allowed till we get rid of all JSON object settings
refs #10318
Because settings are not populated with the correct group and flags, we
must _always_ set these. Then we can check to see if there are values
which need migrating, and if not, can safely exit and leave the values
as default.
refs #10318
If migrating from a previous version that does not include the setting
being migrated from we can safely not update the new setting, and just
rely on its default value being present. When rolling back we can use
defaults if the new setting does not exist.
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates the type field for a setting to specify its type explicitly
- Type can be one of (string, number, boolean, array)
- `object` is allowed a temp type till all the object settings are removed
- existing `type` value for setting was misleading and referring to `group`
- we moved the current `type` value for a setting to `group` here - 4a9e57c170
* Updated members default settings
ref #10318
This pulls out the members_subscription_settings & stripe_connect_intgration settings into separate keys
* Updated usage of members_from_address
* Updated stripe_connect usage
* Updated members config to use new settings
* Updated members middleware to use isStripeConnected
* Updated members service to reload correctly
We reload the members-api instance when the related settings change, so
this makes sure we're listening to the correct settings changes
* Updated ghost_head helper to use new settings
* Updated theme middleware to use new settings
* Renamed members_allow_signup -> members_allow_free_signup
* Fixed tests after settings refactor
* Removed from direct key settings key
* Fixed regression tests for settings api
refs #10318
As populateDefaults is run _before_ migrations, the new settings will
already be inserted in the database, so we just need to update their
values and then delete the old settings.
refs https://github.com/TryGhost/Ghost/issues/10318
`Settings.populateDefaults()` is run before migrations during Ghost's startup. This can cause problems when new settings table columns are added (and populated in `default-settings.json`) because `populateDefaults()` was using the model layer which assumes that those columns are available, resulting in `ER_BAD_FIELD_ERROR: Unknown column` type errors.
- query the database for the available `settings` table columns
- switch to using raw knex queries without Bookshelf for insertions so that we're in control of the columns that are added
- use `_.pick` to skip any properties in `default-settings.json` that do not match to an available column - those columns will be added and populated by later migrations
- moving away from using the model to insert settings has the side-effect of not emitting `settings.added/edited` and `settings.x.added/edited` events, this should be fine because `populateDefaults()` is called before anything else is set up and listening
- added a call to `populateDefaults()` in our knex-migrator "before migration" hook so that we have consistent db state across both startup initialised migrations and manually triggered knex migrations
no-issue
These utils are the first steps toward getting the models out of our
migrations! The utils here interact directly with the database and where
possible don't reinvent the wheel, by using smaller building blocks to
build more comples ones.
refs https://github.com/TryGhost/Ghost/issues/10318
- `shared_views` is always an array (Ghost-Admin has handling to revert it to an array if it's anything else) but it's default value was set to an empty object
refs https://github.com/TryGhost/Ghost/issues/10318
- maps old `settings.type` values to new `settings.type/group` values
- uses an explicit map so that we don't lose information and can safely roll back even though we're modifying `settings.type` too
- updates `settings.type` values too to keep code working while we switch to using `settings.group`
- sets the `settings.group` value for all settings which are keeping the same group as their current type
- adapts `settings.type` validations to match new groups
- adds flags to specific settings, both in the migration for existing settings records and in default-settings.json for new settings records
refs https://github.com/TryGhost/Ghost/issues/10318
- `group`
- to replace the `type` column, provides a more descriptive name for the columns use
- for existing sites it will be populated by migrating data from the `type` column in a later migration
- for new sites a minimal update has been added to `parseDefaultSettings()` to populate the `group` field when settings are created during startup - fixes the NOT NULL constraint on `settings.group`
- `flags`
- signifies special handling that is different to other settings in a group
- eg, `PUBLIC,RO` would indicate that the setting is available via unauthenticated endpoints and is read-only
refs #10318
refs 2614565d5a
- Adds importer mapping for fields as in referenced migration
- The intention is to allow exports from v2/v3 to still be compatilbe with current version
- Adds a mapper for deprecated fields and imports them with new settings "keys"
refs #10318
refs 2614565d5a
- Renamed ghost_head/ghost_foot in settings to match the new names
introduced in migrations
- Above change lead to reshufling in the mappings in input/output
serializers
- Makes sure change is compatible with v2 API
refs https://github.com/TryGhost/Ghost/issues/10318
refs 2614565d5a
- Renames to match referenced migration renames
- Fixed API responses so they are consistent with newly renamed fields
- Not returning lang and timezone keys from settings in API v2 ther rest should be returned in API v3/canary
refs https://github.com/TryGhost/Ghost/issues/10318
- precursor to migrating from `settings.type` to `settings.group`
- renames `blog` type to `site`
- renames `bulk_email` type to `email`
- moves settings out of `site` (previously `blog`) into more appropriate groups such as `core` or individual feature groups
no issue
- Previous migration in commit - 2614565d5a - was using incorrect json structure for brand -> primary_color
- Fixes the up/down migrations to work on correct existing `brand: primary_color` structure
refs https://github.com/TryGhost/Ghost/issues/10318
- Adds following renames to settings table keys:
'default_locale' -> 'lang'
'active_timezone' -> 'timezone'
'ghost_head' -> 'codeinjection_head'
'ghost_foot' -> 'codeinjection_foot'
'brand.publicationColor' -> 'accent_color'
- The renames are done to match revised naming conventions and naming
exposed through APIs
- Supersedes this revert - 1eeb5a60b8 and #11946
refs https://github.com/TryGhost/Ghost/issues/10318
- Adds following renames to settings table keys:
'default_locale' -> 'lang'
'active_timezone' -> 'timezone'
'ghost_head' -> 'codeinjection_head'
'ghost_foot' -> 'codeinjection_foot'
'brand.publicationColor' -> 'accent_color'
- The renames are done to match revised naming conventions and naming
exposed through APIs
refs #10318
refs https://github.com/TryGhost/Ghost/pull/11942
- Removes force_i18n, permalinks, and members_session_secret usage from the codebase
- We deprecated these flasgs and have not used since Ghost v2. It's good time to remove them before we introduce bigger changes to how `settings` table opeartes.
- Fixed importer test. The test was meant to check if string values were converted properly, the check agains boolean didn't make much sense in this context, so removed it.
- Following this change are going to come ralated migrations to fix existing data (see ref)
no issue
- reverts commit 87c31444fd but with modifications to settings naming
- Adds new settings for members modal customization to default settings
- `portal_button` controls the visibility of beacon in members modal
- `portal_name` controls the visibility of name field in signup
- `portal_plans` controls the visibility of plans allowed for member to signup with
- Adds stripe connect check to determine if stripe is setup or not
- Adds the 3 new settings to members site data
- Updates to snake case naming for members site API data
no issue
- Adds new settings for members modal customization to default settings
- `membersjs_show_beacon` controls the visibility of beacon in members modal
- `membersjs_show_signup_name` controls the visibility of name field in signup
- `membersjs_allowed_plans` controls the visibility of plans allowed for member to signup with
- Adds stripe connect check to determine if stripe is setup or not
- Adds the 3 new settings to members site data
refs https://github.com/TryGhost/Ghost/issues/11414
- Importing data currently overwrites the existing "from address" with new value
- "from address" needs to go through email validation flow before update which was bypassed
- Updates importer to not allow overwrite for "from address" and use existing
- Adds test for "from address" overwrite
refs https://github.com/TryGhost/Ghost/issues/11414
We want to allow adding custom domains to member's from address as long as the new email is verified using the magic link flow and not saved directly to DB. Since we currently don't store domain as part of fromAddress, this PR -
- adds migration to update all existing fromAddress by appending site domain
closes https://github.com/TryGhost/Ghost/issues/11841
- Migration adds mapping between permissions and roles for email_preview send test mail
- Only owner previously had correct permission to send test emails
- Fixture existed to allow Admin/Editor/Integrations to send test mails but had missing migration
- Adds tests for roles to send test email
- Represents that logging is shared across all parts of Ghost at present
* moved core/server/lib/common/logging to core/shared/logging
* updated logging path for generic imports
* updated migration and schema imports of logging
* updated tests and index logging import
* 🔥 removed logging from common module
* fixed tests
* moved `server/config` to `shared/config`
* updated config import paths in server to use shared
* updated config import paths in frontend to use shared
* updated config import paths in test to use shared
* updated config import paths in root to use shared
* trigger regression tests
* of course the rebase broke tests
* refactored `core/frontend/apps` to destructure common imports
* refactored `core/frontend/services/{apps, redirects, routing}` to destructure common imports
* refactored `core/frontend/services/settings` to destructure common imports
* refactored remaining `core/frontend/services` to destructure common imports
* refactored `core/server/adapters` to destructure common imports
* refactored `core/server/data/{db, exporter, schema, validation}` to destructure common imports
* refactored `core/server/data/importer` to destructure common imports
* refactored `core/server/models/{base, plugins, relations}` to destructure common imports
* refactored remaining `core/server/models` to destructure common imports
* refactored `core/server/api/canary/utils/serializers/output` to destructure common imports
* refactored remaining `core/server/api/canary/utils` to destructure common imports
* refactored remaining `core/server/api/canary` to destructure common imports
* refactored `core/server/api/shared` to destructure common imports
* refactored `core/server/api/v2/utils` to destructure common imports
* refactored remaining `core/server/api/v2` to destructure common imports
* refactored `core/frontend/meta` to destructure common imports
* fixed some tests referencing `common.errors` instead of `@tryghost/errors`
- Not all of them need to be updated; only updating the ones that are
causing failures
* fixed errors import being shadowed by local scope
- All var declarations are now const or let as per ES6
- All comma-separated lists / chained declarations are now one declaration per line
- This is for clarity/readability but also made running the var-to-const/let switch smoother
- ESLint rules updated to match
How this was done:
- npm install -g jscodeshift
- git clone https://github.com/cpojer/js-codemod.git
- git clone git@github.com:TryGhost/Ghost.git shallow-ghost
- cd shallow-ghost
- jscodeshift -t ../js-codemod/transforms/unchain-variables.js . -v=2
- jscodeshift -t ../js-codemod/transforms/no-vars.js . -v=2
- yarn
- yarn test
- yarn lint / fix various lint errors (almost all indent) by opening files and saving in vscode
- grunt test-regression
- sorted!
- in 3.13.2 the importer always throws the error "The "path" argument must be of type string. Received an instance of Object"
- this is due to a change in method signature that wasn't accounted for
- added a test to catch similar changes to this code in future
no issue
- moved `mobiledoc.renderers.mobiledocHtmlRenderer` to `mobiledoc.mobiledocHtmlRenderer` so that it's easier for the getter to access the parent objects getters
- removed all tests and dependencies that now live in @tryghost/mobiledoc-dom-renderer
- kept the `mobiledocHtmlRenderer` test because that's testing that we've correctly wired up our cards and atoms and the output is what we expect
no issue
- the blank document we use in Ghost is not specific to the html renderer
- renamed from `structure` to `document` to better represent its intent
- allows for easier extraction of `mobiledocHtmlRenderer`
no issue
- importing an LTS export would cause Ghost to throw a 500 error because
it used InternalServerError.
- an IncorrectUsageError is more applicable here
- this commit also updates the code comment and error message
- note: removed comment about WP exports because the plugin has been updated
to support the v2 & v3 format
- spotted in Sentry
no issue
- Migrations within a minor have to be named with numbered prefixes like 01-, 02-, 03-.
- These two migrations were merged into master in the same time window which lead to having incorrect naming
- Moved zipFolder to a new package
- also exposing extract-zip from the new package
- new package has the API pre-promisified
- also uses @tryghost/extract-zip instead of extract-zip, which has bugfixes
- Apps are marked as removed in 3.0, never officially launched and have been deprecated for at least 2 years.
- We've slowly removed bits that got in our way or were insecure over time meaning they mostly didn't work
- This cleans up the remainder of the logic
- The tables should be cleaned up in a future major
- Apps are marked as removed in 3.0, never officially launched and have been deprecated for at least 2 years.
- We've slowly removed bits that got in our way or were insecure over time meaning they mostly didn't work
- This cleans up the remainder of the logic
- The tables should be cleaned up in a future major
no-issue
This adds two new endpoints, one at /ghost/.well-known/jwks.json for exposing
a public key, and one on the canary api /identities, which allows the
Owner user to fetch a JWT.
This token can then be used by external services to verify the domain
* Added ghost_{public,private}_key settings
This key can be used for generating tokens for communicating with
external services on behalf of Ghost
* Added .well-known directory to /ghost/.well-known
We add a jwks.json file to the .well-known directory which exposes a
public JWK which can be used to verify the signatures of JWT's created
by Ghost
This is added to the /ghost/ path so that it can live on the admin
domain, rather than the frontend. This is because most of its
uses/functions will be in relation to the admin domain.
* Improved settings model tests
This removes hardcoded positions in favour of testing that a particular
event wasn't emitted which is less brittle and more precise about what's
being tested
* Fixed parent app unit tests for well-known
This updates the parent app unit tests to check that the well-known
route is mounted. We all change proxyquire to use `noCallThru` which
ensures that the ubderlying modules are not required. This stops the
initialisation logic in ./well-known erroring in tests
https://github.com/thlorenz/proxyquire/issues/215
* Moved jwt signature to a separate 'token' propery
This structure corresponds to other resources and allows to exptend with
additional properties in future if needed
no issue
- The flag has not been used and can be removed, to make the `members_subscription_settings` JSON record in `settings` table easier to read.
- It used to indicate Stripe configuration being present. Currently that is checked by looking up if Stripe config's `public_token` and `secret_token` values are present (example - https://github.com/TryGhost/Ghost/blob/3.11.0/core/frontend/helpers/ghost_head.js#L54)
refs https://github.com/TryGhost/Ghost/issues/11648
- Removes Stripe plan entries from settings that are not formatted correctly.
- Incorrect formatting was caused by a bug in 3.10.0 Admin-Client where it wasn't able to find complimentary plan. Related fix for this here - 9e7a6b801a
no issue
- Adds 'GET /members/:id/signin_urls' endpoint to Admin API allowing to fetch login URL for member. This URL allows to log in as a member which is useful in situations when you need to impersonate a member (for example to debug some issue they are having)
- Added member_signin_urls permission with migrations. Only the "Owner" user can read "signin_urls" resource. Admin and other users will be denied access
no issue
- 3.6.0 contained incorrect references in the `schema.js` file for the `members_label` table that was added in that version. On MySQL knex created a foreign key constraint for that reference which stopped member labels from being createable
- this fixes the schema file and has a migration to drop and recreate the table. Knex handles removal and addition of foreign keys during table drop/create
no issue
* Updated sendEmailWithMagicLink syntax
* Updated label name selection from theme
* Updated migration version for labels
* Added labels to export/import of members
* Added member labels sanitization for case-insensitive duplicates
* Fixed tests
* Fixed label serialization bug on import
* Bumped @tryghost/members-api to 0.15.0
* Fixed lint
* Cleanup
no issue
Since we added `email_subject` to `posts_meta` table in `3.1`, the migration tries to add `email_subject` column from post table, which does not exist and thus tries adding `undefined` value for column. Since sqlite expects default values while inserting new columns, this breaks any migration directly from `1.x`/`2.x` to 3.x.
The fix adds a default `null` value for any post_schema entry which doesn't has a value.
no issue
- The helper allows generating HTML needed to cancel or continue the member's subscription depending on subscription state.
- Added public members endpoint to allow updating subscription's `cancel_at_period_end` attribute available at: `PUT /api/canary/members/subscriptions/:id/`
- Added client-side hook to allow calling subscription cancellation. Allows to create elements with `data-members-cancel-subscription` / `data-members-continue-subscription` attributes which would call subscription update.
- Updated schema and added migration for `current_period_end` column
- As discussed we only add a single column to subscriptions table to avoid preoptimizing for future cases
- Added {{cancel_link}} helper
- Added error handling for {{cancel_link}} when members are disabled
- Added test coverage for {{cancel_link}} helper
- Bumped @tryghost/members-api version to 0.10.2. Needed to use `updateSubscription` middleware
- Bumped gscan to 3.2.0. Needed to recognize new {{cancel_link}} helper
no issue
- Secondary navigation means most nav concepts are supported, e.g. header & footer, or left & right
- The UI is added separately, this PR adds supporting concepts:
- make sure the default value is an empty array
- add support in the API (v3 only)
- add handling in the navigation helper
no issue
- additional migration for the column added since the last 3.1 beta release to allow beta upgrades without rollbacks
- will be a no-op for upgrades from 3.0 as it's covered by `3.1/05-add-emails-table.js`
We want to allow admin users to trigger a retry of failed emails without having to go through the unpublish/republish dance.
- fixed resource identifier in email permissions migration so email permissions are added correctly
- added new email permissions migration so that beta releases can be upgraded without rollback (will be a no-op for any non-beta upgrades)
- added `/emails/:id/retry/` canary Admin API endpoint
- follows same URL pattern as theme activation
- only triggers mega service retry endpoint if the email has a `'failed'` status
no issue
- Increased default mailgun retry limit to 5
- Handling retry logic closer to SDK layer gives less future manual handling
- Allowed failing request to be passed through to the caller
- To be able to handle failed requests more gracefully in the future we need all available error information to be given to the caller
- The previous method with `Promise.all` would have rejected a whole batch without providing details on each specific batch.
- Limited data returned with a failed message to batch values
- Added better error handling on mega layer
- Added new column to store failed batch info
- Added reference to mailgan error docs
- Refactored batch emailer to respond with instances of an object
- It's hard to reason about the response type of bulk mailer when multiple object types can be returned
- This gives more clarity and ability to check with `instanceof` check
no issue
- The limitation on Mailgun side of API seems to be 1000 emails per message.
- The only place where I could find a hard limit of 1000 emails per
batch was this PHP SDK issue: https://github.com/mailgun/mailgun-php/issues/469
- To store ids of sent messages introduce a mega column on the emails table. They can be synced with stats or other metrics during even pooling in the future
- Removed redundant `join(',')` statement.The SDK accepts an array of emails as well. Less code - better code :)
no issue
- Fixed default email property output when the empty value is returned
- This is needed for consistency with other endpoint properties like primary_tag which are null when there is no value assigned
- Updated acceptance tests to handle email property
- Schema had to be updated to not use reference so that the information about email can be independent of the post - can still exist if the post is deleted
- Renames were done as that suits how emails would be handled by the bulk email handler
- These statuses are only for internal representation of the state and don't represent what happens to emails delivery-wise
- There is no need for 'sent' status as emails are "never done" and stats wold be checked for stats field would be used to check on details of the status
refs https://github.com/TryGhost/Ghost/pull/11270
- Fixed 3.0/11-update-posts-html migration which failed in scenario when more than 999 posts with posts_meta relation were present
- The issue was originally spotted here: https://github.com/TryGhost/Ghost/pull/11270#issuecomment-546248308
- The main problem is in the `SELECT` statement which is generated for `findAll` method in Bookshelf which creates `WHERE IN(post_ids_here)` statement with all posts in the database
- Using knex directly as that's a preferred way to write migrations (does not depend on the model layer)
closes#11263
- Fixed `3.0/05-populate-posts-meta-table.js` migration failure when having >999 posts with metadata in the database
- The issue here is with hitting SQLite's internal SQLITE_LIMIT_VARIABLE_NUMBER limit when updating with a large amount of posts having metadata fields set (ref.: https://sqlite.org/limits.html#max_variable_number)
- Transforming migration to iterative method avoided inserting lots of records at once
no issue
We split `posts` table into 2 in v3 with a new `posts_meta` table. Since migrations always use the version of code which is being migrated to - in this case the Post model - which in v3 relies on the posts_meta table, `2.x` migrations relying on post model will fail as it doesn't exist in the expected state. This PR updates all 2.x migrations using `models.Post` to use knex queries directly to access database and perform operations.
no issue
We added 2 new member subscription settings - `allowSelfSignup` and `fromAddress`- with defaults as `true` and `noreply`, this migration sets default values for both settings for users migrating from previous version and cleans up intermediate naming for `allowSelfSignup`.
no issue
Since we removed subscribers code in v3, we cannot use `models.Subscribers` for migration, and instead switch to using db directly for fetching existing subscribers before migrating them to members.
refs https://github.com/TryGhost/Ghost/pull/11152
- Added subscribers table drop migration
- Removed subscribers from schema
- Removed subscribers controllers/routes/regression tests
- Removed subscriber related API code
- Removed subscribers from internal apps
- Removed subscriber importer
- Removed subscriber model
- Removed subscriber related permissions
- Removed webhook code related to subscribers
- When upgrading to v3 it is on the site admin to migrate all zapps or any other webhook clients to use members
- Removed subscriber-specific translation
- Removed subscriber lab flag
* Added new `requirePaymentForSignup` setting for members
no issue
- Adds new `requirePaymentForSignup` setting flag for members, `false` by default.
- Wired members API `allowSelfSignup` to `requirePayment` setting
closes#11207
MySQL doesn't allow unqiue keys with a length of more than 191 when using InnoDB with utfmb4. These changes will ensure any incorrect tables created are fixed and have the
correct length for customer_id
* Changed `customer_id` to non-unique column
* Nooped the 2.32 `members_stripe_customers` migration
* Added migration to recreate `members_stripe_customers` table
* sqlite doesn't allow `ALTER TABLE` queries so this is the cleanest solution considering the table is not yet in use
no issue
- Moved default_content_visibility out of labs as we should be extra careful with what is exposed in the labs + it doesn't really belong there.
no issue
- Removed uses of `visibility` column in frontend url service configs
- The value of `visibility` is always set to 'public' in posts at the moment and doesn't serve any specific purpose when used with these filters.
- Allowed new visibility attributes in post model
- `posts.visibility` column is being repurposed for the needs of member content gating
- Added test for visibility editing in Admin API
- Corrected test schema checks for Admin API post/page responses
no issue
- Populates members table with existing subscribers. Only takes into account columns we know already exist and need to be copied i.e `name`/`email`
no-issue
This will be used to store stripe specific information for members
customer_id has a max length of 255 https://stripe.com/docs/upgrades
member_id is not unique as we cannot ensure that a member doesn't have
more than one customer object associated with them. e.g. if they signup
twice, or if they cancel, and signup again, creating a new customer.
We probably won't handle this case to begin with, but we will keep the
data intact.
no issue
- bumps `knex-migrator` so it supports irreversible migrations
- marks the `03-drop-client-auth` migration as irreversible because it destroys data that is not recoverable and is required for earlier versions of Ghost to function
no issue
- rollbacks have switched to using transactions but the migration code was copied from an old migration coded before that switch
- `down()` is no longer called with an object that contains a `connection` key, it has `transacting` instead
no issue
- `knex-migrator` will run migrations in the order that nodejs provides when running `fs.readDirSync` which in most cases is strict alphabetical
- 3.0 will shortly have more than 9 migrations which was resulting in the migration order being 1, 10, 2
- prefixing all single-digit migrations with `0` means that strict alphabetical ordering results in the expected order
refs https://github.com/TryGhost/Ghost/issues/10922
- adds migrations to...
1. add `post.type` column
2. populate `post.type` column based on `post.page` value
3. drop `post.page` column
- updates all code paths to work with `post.type` in place of `post.page`
- adds `nql-map-key-values` transformer for mapping `page`->`type` in `filter` params when using the v2 API
- modifies importer to handle `post.page`->`post.type` transformation when importing older export files
NOTE: The post metadata table split is purely an internal optimization for v3 and doesn't require or expect any external actions including related API usage in v3
We keep running into issues adding new fields to the post table because there are too many fields making the post table "too wide". We have also hit MySQL limitations in how many bytes can be in a row (64kb) with post table.
In v3, we decided to split the 8 post fields (meta, twitter and og) used for meta data into a posts_meta table as these 8 fields are all "problem" `varchar` fields and make sense logically grouped together. The API layer is unaffected by the split as input/output serializers ensure the data flow works the same way as it was in v2. Only thing to note is json export in v3 will have slightly different structure with posts meta fields as separate.
- Creates new post_meta schema/table with 8 fields (2 meta_* , 3 twitter_* and 3 og_*)
- Update relations between post and post_meta table
- Update input/output serializers to keep existing API behavior
- Avoids new entry in post_meta table for post where all meta fields are null
- Keeps the current fields API param behavior
- Handles migration of existing posts to new table structure
- Updates importer/exporter to work seamlessly with table changes
no issue
- drops now-unused `accesstokens`, `refreshtokens`, `clients`, and `client_trusted_domains` tables
- no rollback because the db schema for the tables no longer exists
no issue
- v0.1 is ☠️ so the access/refresh token based auth is no longer used
- removed all code related to the `accesstokens` and `refreshtokens` tables
- removed all `passport` related dependencies as it's no longer used
no issue
- v0.1 is ☠️ so there's no longer any use of client auth
- removes all code related to `clients` and `client_trusted_domains`
- noops the "add backup client" migration in 1.7 because the referenced fixture no longer exists causing migrations and consequently all regression tests to fail
no issue
- Drops `ghost_auth_access_token` and `ghost_auth_id` fields since not used anymore
- Adds migration for dropping these columns from users table
- Drops Auth strategy - `ghostStrategy` - since its not used anymore
no issue
- we recently started wrapping rollbacks in transactions (https://github.com/TryGhost/knex-migrator/pull/161)
- in a number of migrations we were calling `model.destroy()` without passing through the options which includes the current transaction
- for models which are using `bookshelf-relations` this could result in an internal `SQLITE_BUSY: database is locked` error because it tries to run queries against tables that have been locked by previous queries in the transaction
- by passing through the options when calling `.destroy()` it allows the `bookshelf-relations` to re-use the same transction avoiding the database lock problems
no issue
- the column addition/removal can be too slow for large sites
- will be added back in 3.0
---
Revert "Fixed canary api for page/type column"
This reverts commit a5a7e7e919.
Revert "Updated frontend canary url config for page/type"
This reverts commit 19100ec5e6.
Revert "Updated canary api to handle type column correctly (#11006)"
This reverts commit c3e8ba0523.
Revert "Ensured `page` filter works in routes.yaml"
This reverts commit 9037c19e50.
Revert "Replaced usage of mongo util with nql-map-key-values"
This reverts commit 8c5f1d0ef0.
Revert "Added shared nql-map-key-values module"
This reverts commit ef4fd4b8ef.
Revert "Ensured page prop is present on content api response"
This reverts commit cfa0a0862b.
Revert "Fixed failing regression tests"
This reverts commit 9c2bb3811f.
Revert "Updated xmlrpc and slack service to use type column"
This reverts commit 44a02c7d36.
Revert "Updated v0.1 posts api to work with type column"
This reverts commit 2c81d7c914.
Revert "Removed updates to v0.1 specific code"
This reverts commit 08d83c1f53.
Revert "Added missing context from ValidationError"
This reverts commit cd45ab4f54.
Revert "Renamed page->type in the page&posts serializers"
This reverts commit df99e724e3.
Revert "Added mongo helper to input serializers"
This reverts commit fb8eadb4a8.
Revert "Passed mongoTransformer through to NQL"
This reverts commit 0ae3f0fdfc.
Revert "Permitted mongoTransformer option for read methods"
This reverts commit a89376bf26.
Revert "Updated the count plugin to reference the type column"
This reverts commit a52f15d3d3.
Revert "Updated hashes for db integrity check"
This reverts commit bb6b337be3.
Revert "Remove page column and remaining references"
This reverts commit 9d7190d692.
Revert "Added type column to data generator"
This reverts commit e59806cb45.
Revert "Removed references to page column in rss tests"
This reverts commit 04d0f855de.
Revert "Removed page column references in validation tests"
This reverts commit f0afbc5cc0.
Revert "Updated the post model to use the `type` column"
This reverts commit 1189bc823a.
Revert "Updated url service to use type column"
This reverts commit 61612ba8fd.
Revert "Updated the v2 api to deal with type column"
This reverts commit 57afb2de2b.
Revert "Added type property to post model defaults"
This reverts commit dc3345b1c5.
Revert "Added type property to the default post fixtures"
This reverts commit 82d8c38033.
Revert "Added type column to posts table"
This reverts commit 9b85fc6a69.
no issue
- updates the attribute sanitiser of the posts importer to convert `post.page=true/false` to `post.type='page'/'post'
- gives precedence to `post.type` if an imported post somehow has both `post.page` and `post.type` attributes
no issue
- the `mobiledoc_revisions` table can grow very large in certain circumstances which can result in Out-Of-Memory errors when performing backups, resulting in failed upgrades
- adds `mobiledoc_revisions` to the exporter excluded tables list as a temporary solution until we have safer export creation and/or improved revision handling
refs #8717
- The posts without slugs should not be taken into account when detecting duplicates as slug field is not required when importing.
- Ideal solution would require generating slug before duplicate detection phase. This would cause duplicate detection to take 'title' into account which didn't happen before.
refs #10922
When rolling back the removal of the page column, we must re-add it, but
the definition for it has been removed from the schema, so we must
hardcode the definition.
closes#10060
- Implemented scheduling for posts and pages
- Added cache invalidation when scheduling
- Refactored admin token eneration function to accept existing key as parameter in tests
- Added Ghost Scheduler Integration fixture
- Added fixture for permissions for post publish action
- Migrated getScheduled method to v2
- Did not add support for 'from' and 'to' parameters as they were not used by DefaultScheduler
- This method needs rethinking in a long run as it's an ugly hack and should rather become proper endpoint that returns JSON data instead of models
- Removed unused auth middleware from v2 routes
- Added internal scheduler role
- Implemetnted transactions in v2 frame
- This takes into account scenario mentioned in c93f03b87e
- Specifically:
>if two queries happening in a transaction we have to signalise
knex/mysql that we select for an update
otherwise the following case happens:
you fetch posts for an update
a user requests comes in and updates the post (e.g. sets title to "X")
you update the fetched posts, title would get overriden to the old one
closes#10788
This adds an extra filter to the preImport method of the settings
importer to removes settings with the key `is_private`
This message is specifically only for when an import has privacy mode ON
and the current site has privacy mode OFF.
no issue
- we try to store all urls as relative paths where possible in Ghost so that the `config.url` value can be changed
- all relative paths are stored as root-relative except for the `post.canonical_url` field which was storing subdirectory-relative paths
- adds a migration to put the subdirectory prefix onto any relative canonical_url paths
- updates the canonical_url input serialiser to keep the subdirectory rather than stripping it to match all other url fields
* Simplified db controller permissions options
The existing objects were confusing because they did the same thing as
setting permissions to true, but gave the impressions that something
special was happening/required.
* Added DB Backup Integration Role
This will allow us to assign certain api_keys this role, in order to
automate db backups
* Allowed admin api_keys to have configurable roles
This will allow keys for the admin api to do customised things such as db export
* Added ghost-backup integration to fixtures
* Added migrations for DB Backup Integration and role
no-issue
This module was first created (AFAICT) in c09c20ad8d (diff-20a31f345ca2643b2602224678bb8d5b) and
has since undergone some filename renames and eslint refactors - we don't support
PostgreSQL and have no immediate plans to do so.
refs #10921
- New SEO related fields will now be available as a part of Admin API /settings endpoint
- The ordering of fields is taken from post's schema
- Extended settings test suite with new SEO fields
- Adjusted settings model unit test
closes#10785
- The behavior for tags will now be similar to posts' one described in the docs
- "The only strictly required field when importing posts is the title. Ghost will automatically generate slugs and set every other field to the default or empty."
- The breaking change was introduced with: 68d8154d4f (diff-e712df50c0dc7cf33746eeff0564003cR97) (assumed there's always slug in the imported object which is not true)
- Added originalIdMap to the importer base class to track id
substitution so it can be used when dealing with relational resource
updates
- Removed explicit use of 'this.stripProperties(['id']);' in
beforeImport of base class because we need to assign and remove the id
property in the same place to track this change
- Only calling 'this.stripProperties(['id']);' in
settings/trusted_domain imports as the method won't be called otherwise
- Expanded regression tests with new supported import case
no issue
- Updated Test & linting packages
- Updated use of hasOwnProperty
- Using Object.prototype.hasOwnProperty instead (ref. eslint.org/docs/rules/no-prototype-builtins)
- Removed already defined built-in global variable Intl
- Applied `--fix` with lint command on `core/test` folder
- The rules were broken because some of them were made stricter for `eslint: recommended` ruleset (ref. https://eslint.org/docs/user-guide/migrating-to-6.0.0#eslint-recommended-changes)
- Removed redundant global variable declarations to pass linting
refs #10790
- The code was moved out of controllers to reduce the number of coupling points between the API controllers and "frontend" services
- A nice side effect of this move is a decreased amount of code that will need to be maintained and reusability between existing controllers
- Calling just a few methods from frontend services on API level makes it easier to abstract fronted away from API
refs #10790
- Moved /core/apps into core/frontend
- Moved /core/server/helpers to /core/frontend/helpers along with /core/server/services/themes
- Changed helper location in overrides
- Moved /core/server/services/routing to /core/frontend/services
- Moved /core/server/services/url to /core/frontend/services
- Moved /core/server/data/meta to /core/frontend/meta
- Moved /core/server/services/rss to /core/frontend/services
- Moved /core/server/data/xml to /core/frontend/services
closes#10773
- The refactoring is a substitute for `urlService.utils` used previously throughout the codebase and now extracted into the separate module in Ghost-SDK
- Added url-utils stubbing utility for test suites
- Some tests had to be refactored to avoid double mocks (when url's are being reset inside of rested 'describe' groups)
refs #10618
- Visibility methods don't belong on model, but are generic utils
- Used directly from ghost helper's visibility methods, cleans up core
- Removes direct model dependency of theme helper
- Updated `foreach_spec` to correct test data as per schema - visibility property cannot be empty
refs https://github.com/TryGhost/Ghost/issues/9414
refs c9b95b4bbd
- Removed package version from asset hash calculation
- Package version doesn't introduce any value when calculating a hash because Date.now() provides enough randomization on its own
closes#10706
- Fixed a bug where we weren't assigning `summary_large_image` as Twitter card property when no feature image is provided, but a dedicated Twitter image
- Updated test to reflect this case to be tested better
no-issue
It turned out that due to the mismatch between `"members"` and
`"member"`, that not a single row was added to the database via this
migration. Clearing the file for simplicity.
no-issue
The original migration read data from the fixtures, and ensured the
relations were made. Now the fixtures have been corrected, we can use
the same migration for 2.22
no-issue
The previous relations were setup to match against "members", which is
plural, but the object_type on the permission is "member". This was
causing the permissions to not be added as a relation to the role.
