refs https://github.com/TryGhost/Ghost/issues/12602
* Updated members_status_events table
By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range
* Updated MEGA to handle addition of 'comped' status
With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"
* Updated customQuery for MemberStatusEvent
With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.
* Populated members_status_events table
As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.
The assumptions used in backfilling is that a Member's current status,
is their only status.
refs https://github.com/TryGhost/Ghost/issues/10318
- Object format used in previous "slack" setting was considered an
anti-pattern. Flag structure of separate slack_url and slack_username
values was extracted out of the "slack" JSON.
refs https://github.com/TryGhost/Ghost/issues/12602
As we only want to add events for when an email is changed, we need to store the previous email if we want a complete log of all emails for a member
refs https://github.com/TryGhost/Ghost/issues/12602
* Added members_payment_events table
This table will store successful and unsuccessful payment attempts, and
can be used to calculate gross volume over time.
* Added members_login_events table
This table can be used to audit member logins
* Added members_email_change_events table
This table will allow us to store a history of email addresses associated with a member
* Added members_status_events table
This table will allow us to track the change in status over time for members, as well
as calculate aggregates over time, e.g. paid members over time
* Added members_paid_subscription_events
This table will allow us to track subscriptions changes for members, as well as
calculating MRR over time
refs https://github.com/TryGhost/Ghost/issues/12567
- Changing unique constraint from slug to slug+type should allow for posts and pages to be created with the same slug
- The constraint will be present on application layer for API v4 while we figure out how to deal with it in API v5
refs https://github.com/TryGhost/Ghost/issues/12567
- The method was using unneeded lodash dependency and was too complex for what it was doing
- Reshuffled internal code to use native JS filter/forEach iterators
refs https://github.com/TryGhost/Ghost/issues/12567
- Introduced here @@UNIQUE_CONSTRAINTS@@ notation allows to create unique contraints over multiple database fields. This will be needed to change posts' table unique constraint from `slug` to `slug+type`.
- The notation is equivalent to SQL's: UNIQUE(column_name1,column_name2)
- Example use in schema:
posts: {
slug: {type: 'string', maxlength: 191, nullable: false},
type: {type: 'string', maxlength: 50, nullable: false, defaultTo: 'post', validations: {isIn: [['post', 'page']]}},
'@@UNIQUES@@': [
['slug', 'type']
]
}
no-issue
This value is nullable from the Stripe API so we're making sure that we
can store exact values locally.
SQLite3 does not supports altering columns so instead we have to:
1. Create a temp table and copy the data to it
2. Delete the original table
3. Recreate the original table with the necessary modifications
4. Copy the data from the temp table
5. Drop the temp table
refs #12160
This flag will allow us easier filtering of members via the API
* Added status column to members table
This flag will be used to determine if a member is free or paid, rather
than relying on joins with the customers and subscriptions tables.
* Added migration to populate members.status
As we add the column with a default value of "free" we only need to care
about the paid members here. We also preemptively handle migrations for
SQLite where there are > 998 paid members.
no refs
Adds new FirstPromoter settings similar to amp, which allows sites to take advantage of FirstPromoter to launch their own member referral program natively.
- Adds new firstpromoter settings group
- Adds `firstpromoter` setting to group
- Adds `firstpromoter_id` setting to group for FirstPromoter referral tracking id
- Updated tests
closes#12181
refs https://github.com/TryGhost/Ghost/pull/12265
- The 'core' value is invalid for settings 'type' column
- The 'core' default value for 'type' column in model is also invalid
- Both need to be removed as they are never used and only introduce confusion into the codebase
refs https://github.com/TryGhost/Ghost/issues/12461
- adds `members.email_count` and `members.email_opened_count` columns to contain cached counts for faster queries when outputting member data via API
- adds migration to populate cached counts with existing data
- tested locally on ~50k members which took ~4sec on mysql
- updates members output serializer to include the new fields in API output
refs https://github.com/TryGhost/Ghost/issues/12421
- nullable so we can distinguish between members that have and haven't received any trackable emails
- indexed because we'll be using this column for sorting
no issue
- cleans up unused tables `emails.{meta,stats}`
- adds timestamp columns `email_recipients.{delivered_at,opened_at,failed_at}` that can be used for event timelines and basic stats aggregation
- indexed because we want to sort by these columns to find the "latest event" when limiting Mailgun events API requests
- adds aggregated stats columns `emails.{delivered_count,opened_count,failed_count}`
- adds a composite index on `email_recipients.[email_id,member_email]` to dramatically speed up `email_recipient` update queries when processing events
- modifies the db initialisation to support an `'@@INDEXES@@'` key in table schema definition for composite indexes
- these helpers remove a lot of the duplicated code that we had when doing up/down column migrations, and provides a much shorter way of doing this in the future
no issue
Updated newsletter badge to be shown by default which was previously set to be hidden. Users can still control the visibility via Email settings in Ghost-Admin
no-issue
We want to give users to ability to customise the content of their newsletter, and the first step
toward that is a setting in which we can store text or html to embed in the template
no-issue
* Handled send_email_when_published in Posts API
This restores backwards compatibility of the Posts API allowing existing
clients to continue to use the `send_email_when_published` flag. This
change uses two edits, which is unfortunate. The reason being is that
this is an API compatibility issue, not a model issue, so we shouldn't
introduce code to the model layer to handle it. The visibility property
of the model is used to determine how to fall back, and because it can
be left out of the API request, and relies on a default in the settings,
we require that the model decide on the `visibility` before we run our
fallback logic (or we duplicate the `visibility` default at the cost of
maintenance in the future)
* Dropped send_email_when_published column from posts
Since this column is not used any more, we can drop it from the table.
We include an extra migration to repopulate the column in the event of
a rollback
* Updated importer to handle send_email_when_published
Because we currently export this value from Ghost, we should correctly
import it. This follows the same logic as the migrations for this value.
* Included send_email_when_published in API response
As our v3 API documentation includes `send_email_when_published` we must
retain backward compatibility by calculating the property.
* Fixed fields filter with send_email_when_published
* Added safety checks to frame properties
Some parts of the code pass a manually created "frame" which is missing
lots of properties, so we check for the existence of all of them before
using them.
* Fixed 3.1 migration to include columnDefinition
We require that migrations have all the information they need contained
within them as they run in an unknown state of the codebase, which could
be from the commit they are introduced, to any future commit. In this
case the column definition is removed from the schema in 3.38 and the
migration would fail when run in this version or later.
refs https://github.com/TryGhost/Ghost/issues/12355
- Adds new default settings for newsletter customisations - `newsletter_show_badge`, `newsletter_show_header` and `newsletter_body_font_category`
- Adds migrations to update group for new settings
- Add migration to update settings based on existing config value for newsletter settings
- Passes new newsletter settings to newsletter template and updates design based on them
- Fix tests
no issue
- tracking of bulk email opens can be enabled/disabled over time, if we're calculating analytics for emails we don't want emails which didn't have tracking enabled skewing the results so we need a record of whether tracking was enabled for each email
no-issue
This column will allow us to store the canonical recipient filter on the
email resource giving us a detailed log of which members an email was
intended for
no-issue
This column will allow us to decouple the recipients of newsletter from
the `visibility` of a post, allowing us to send emails to specifically
free members, or to send paid posts as newsletters to all members.
no issue
- all staff users can browse/read snippets so they're usable in the editor for everyone
- only administrators, editors, and admin integrations are able to create/edit/delete snippets
no issue
- minimal table structure required for the first iteration of content snippets
- snippets are stored pieces of re-usable content that could effectively be entire posts so the `mobiledoc` field length matches the `posts.mobiledoc` field length
no issue
The email table should be a reference for all data that was used when sending an email. From and Reply-to addresses can change over time and we don't have any other reference for their value at the time of sending an email so we should store them alongside the email content.
- schema updated with `from` and `reply_to` columns
- both are set to `nullable` because we don't have historic data (can be populated and changed in later migrations if needed)
- neither `from` or `reply_to` have `isEmail` validations because they can have name+email in an email-specific format
- will help keep concerns separated in the future. `mega` service can deal with all of the email contents/properties, and the `bulk-email` service's concerns are then only email sending and any provider-specific needs
no-issue
After discussion with Matt, we decided that 192 bits for the token is a
good number, as it has no padding when base64 encoded and is more secure
than 128 bits, whilst still a managable size.
no-issue
This is a table to store single use tokens for use in magic links, the
columns are as simple as possible at the moment and are designed as:
id - standard ObjectID like all of our tables
token - 128bit base64 encoded string
data - arbitrary data to store against the token
created_at - timestamp to allow for expiry to be implemented for tokens
no issue
We want to store a list of recipients for each bulk email so that we have a consistent set of data that background processing/sending jobs can work from without worrying about moving large data sets around or member data changing mid-send.
- `email_batches` table acts as a join table with status for email<->email_recipient
- stores a provider-specific ID that we get back when submitting a batch for sending to the bulk email provider
- `status` allows for batch-specific status updates and picking up where we left off when submitting batches if needed
- explicitly tying a list of email recipients to a batch allows for partial retries
- `email_recipients` table acts as a join table for email<->member
- `member_id` does not have a foreign key constraint because members can be deleted but does have an index so that we can efficiently query which emails a member has received
- stores static copies of the member info present at the time of sending an email for consistency in background jobs and auditing/historical data
refs #11999
- The `routes_hash` setting will be used during the boot process to update the hash
of currently loaded routes.yaml file in case it's different from last restart
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account, public setting
- `members_reply_address` - Where you receive responses to newsletters
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account
- `members_reply_address` - Where you receive responses to newsletters
refs #12126
- Adds migration to add impersonation permission to administrators
- Adds default permission fixture to allow administrators to read member impersonation urls
- Allows administrators to create member impersonation magic links
- deleted files under `core/server/lib/promise` and related test files
- added `@tryghost/promise` as a dependency
- fixed all local requires to point to the new package
no-issue
We are in the process of creating migrations to add foreign key constraints
and cascading deletes to the members_stripe_* tables to make listing members
and deleting members faster. As well as the migrations we need to update the
database schema so that new installations have the correct indexes and constraints.
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
no-issue
They will be used to store webhook information so that we can persist it between
boots and simplify the creation process of webhooks in members
no issue
- Adds new portal settings - `portal_button_style`, `portal_button_icon` and `portal_button_signup_text`
- New settings allows customization of portal button
- Updates tests to include new settings
no issue
- The membersjs default trigger button was set to be visible by default, even behind dev flag
- This changes the default visibility to false so it needs to be switched on from Admin to be visible on screen
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates default settings to contain correct type and validation for each setting
- Updates `populateDefaults` to correctly insert type value for new settings
- Updates settings schema to allow only select types - `array`, `number`, `boolean`, `string`
- `object` is a temporary type allowed till we get rid of all JSON object settings
* Updated members default settings
ref #10318
This pulls out the members_subscription_settings & stripe_connect_intgration settings into separate keys
* Updated usage of members_from_address
* Updated stripe_connect usage
* Updated members config to use new settings
* Updated members middleware to use isStripeConnected
* Updated members service to reload correctly
We reload the members-api instance when the related settings change, so
this makes sure we're listening to the correct settings changes
* Updated ghost_head helper to use new settings
* Updated theme middleware to use new settings
* Renamed members_allow_signup -> members_allow_free_signup
* Fixed tests after settings refactor
* Removed from direct key settings key
* Fixed regression tests for settings api
refs https://github.com/TryGhost/Ghost/issues/10318
- `shared_views` is always an array (Ghost-Admin has handling to revert it to an array if it's anything else) but it's default value was set to an empty object
refs https://github.com/TryGhost/Ghost/issues/10318
- maps old `settings.type` values to new `settings.type/group` values
- uses an explicit map so that we don't lose information and can safely roll back even though we're modifying `settings.type` too
- updates `settings.type` values too to keep code working while we switch to using `settings.group`
- sets the `settings.group` value for all settings which are keeping the same group as their current type
- adapts `settings.type` validations to match new groups
- adds flags to specific settings, both in the migration for existing settings records and in default-settings.json for new settings records
refs https://github.com/TryGhost/Ghost/issues/10318
- `group`
- to replace the `type` column, provides a more descriptive name for the columns use
- for existing sites it will be populated by migrating data from the `type` column in a later migration
- for new sites a minimal update has been added to `parseDefaultSettings()` to populate the `group` field when settings are created during startup - fixes the NOT NULL constraint on `settings.group`
- `flags`
- signifies special handling that is different to other settings in a group
- eg, `PUBLIC,RO` would indicate that the setting is available via unauthenticated endpoints and is read-only
refs #10318
refs 2614565d5a
- Renamed ghost_head/ghost_foot in settings to match the new names
introduced in migrations
- Above change lead to reshufling in the mappings in input/output
serializers
- Makes sure change is compatible with v2 API
refs https://github.com/TryGhost/Ghost/issues/10318
refs 2614565d5a
- Renames to match referenced migration renames
- Fixed API responses so they are consistent with newly renamed fields
- Not returning lang and timezone keys from settings in API v2 ther rest should be returned in API v3/canary
refs https://github.com/TryGhost/Ghost/issues/10318
- precursor to migrating from `settings.type` to `settings.group`
- renames `blog` type to `site`
- renames `bulk_email` type to `email`
- moves settings out of `site` (previously `blog`) into more appropriate groups such as `core` or individual feature groups
refs #10318
refs https://github.com/TryGhost/Ghost/pull/11942
- Removes force_i18n, permalinks, and members_session_secret usage from the codebase
- We deprecated these flasgs and have not used since Ghost v2. It's good time to remove them before we introduce bigger changes to how `settings` table opeartes.
- Fixed importer test. The test was meant to check if string values were converted properly, the check agains boolean didn't make much sense in this context, so removed it.
- Following this change are going to come ralated migrations to fix existing data (see ref)
no issue
- reverts commit 87c31444fd but with modifications to settings naming
- Adds new settings for members modal customization to default settings
- `portal_button` controls the visibility of beacon in members modal
- `portal_name` controls the visibility of name field in signup
- `portal_plans` controls the visibility of plans allowed for member to signup with
- Adds stripe connect check to determine if stripe is setup or not
- Adds the 3 new settings to members site data
- Updates to snake case naming for members site API data
no issue
- Adds new settings for members modal customization to default settings
- `membersjs_show_beacon` controls the visibility of beacon in members modal
- `membersjs_show_signup_name` controls the visibility of name field in signup
- `membersjs_allowed_plans` controls the visibility of plans allowed for member to signup with
- Adds stripe connect check to determine if stripe is setup or not
- Adds the 3 new settings to members site data
- Represents that logging is shared across all parts of Ghost at present
* moved core/server/lib/common/logging to core/shared/logging
* updated logging path for generic imports
* updated migration and schema imports of logging
* updated tests and index logging import
* 🔥 removed logging from common module
* fixed tests
* refactored `core/frontend/apps` to destructure common imports
* refactored `core/frontend/services/{apps, redirects, routing}` to destructure common imports
* refactored `core/frontend/services/settings` to destructure common imports
* refactored remaining `core/frontend/services` to destructure common imports
* refactored `core/server/adapters` to destructure common imports
* refactored `core/server/data/{db, exporter, schema, validation}` to destructure common imports
* refactored `core/server/data/importer` to destructure common imports
* refactored `core/server/models/{base, plugins, relations}` to destructure common imports
* refactored remaining `core/server/models` to destructure common imports
* refactored `core/server/api/canary/utils/serializers/output` to destructure common imports
* refactored remaining `core/server/api/canary/utils` to destructure common imports
* refactored remaining `core/server/api/canary` to destructure common imports
* refactored `core/server/api/shared` to destructure common imports
* refactored `core/server/api/v2/utils` to destructure common imports
* refactored remaining `core/server/api/v2` to destructure common imports
* refactored `core/frontend/meta` to destructure common imports
* fixed some tests referencing `common.errors` instead of `@tryghost/errors`
- Not all of them need to be updated; only updating the ones that are
causing failures
* fixed errors import being shadowed by local scope
- All var declarations are now const or let as per ES6
- All comma-separated lists / chained declarations are now one declaration per line
- This is for clarity/readability but also made running the var-to-const/let switch smoother
- ESLint rules updated to match
How this was done:
- npm install -g jscodeshift
- git clone https://github.com/cpojer/js-codemod.git
- git clone git@github.com:TryGhost/Ghost.git shallow-ghost
- cd shallow-ghost
- jscodeshift -t ../js-codemod/transforms/unchain-variables.js . -v=2
- jscodeshift -t ../js-codemod/transforms/no-vars.js . -v=2
- yarn
- yarn test
- yarn lint / fix various lint errors (almost all indent) by opening files and saving in vscode
- grunt test-regression
- sorted!
- Apps are marked as removed in 3.0, never officially launched and have been deprecated for at least 2 years.
- We've slowly removed bits that got in our way or were insecure over time meaning they mostly didn't work
- This cleans up the remainder of the logic
- The tables should be cleaned up in a future major
- Apps are marked as removed in 3.0, never officially launched and have been deprecated for at least 2 years.
- We've slowly removed bits that got in our way or were insecure over time meaning they mostly didn't work
- This cleans up the remainder of the logic
- The tables should be cleaned up in a future major
no-issue
This adds two new endpoints, one at /ghost/.well-known/jwks.json for exposing
a public key, and one on the canary api /identities, which allows the
Owner user to fetch a JWT.
This token can then be used by external services to verify the domain
* Added ghost_{public,private}_key settings
This key can be used for generating tokens for communicating with
external services on behalf of Ghost
* Added .well-known directory to /ghost/.well-known
We add a jwks.json file to the .well-known directory which exposes a
public JWK which can be used to verify the signatures of JWT's created
by Ghost
This is added to the /ghost/ path so that it can live on the admin
domain, rather than the frontend. This is because most of its
uses/functions will be in relation to the admin domain.
* Improved settings model tests
This removes hardcoded positions in favour of testing that a particular
event wasn't emitted which is less brittle and more precise about what's
being tested
* Fixed parent app unit tests for well-known
This updates the parent app unit tests to check that the well-known
route is mounted. We all change proxyquire to use `noCallThru` which
ensures that the ubderlying modules are not required. This stops the
initialisation logic in ./well-known erroring in tests
https://github.com/thlorenz/proxyquire/issues/215
* Moved jwt signature to a separate 'token' propery
This structure corresponds to other resources and allows to exptend with
additional properties in future if needed
no issue
- The flag has not been used and can be removed, to make the `members_subscription_settings` JSON record in `settings` table easier to read.
- It used to indicate Stripe configuration being present. Currently that is checked by looking up if Stripe config's `public_token` and `secret_token` values are present (example - https://github.com/TryGhost/Ghost/blob/3.11.0/core/frontend/helpers/ghost_head.js#L54)
no issue
- Adds 'GET /members/:id/signin_urls' endpoint to Admin API allowing to fetch login URL for member. This URL allows to log in as a member which is useful in situations when you need to impersonate a member (for example to debug some issue they are having)
- Added member_signin_urls permission with migrations. Only the "Owner" user can read "signin_urls" resource. Admin and other users will be denied access
no issue
- 3.6.0 contained incorrect references in the `schema.js` file for the `members_label` table that was added in that version. On MySQL knex created a foreign key constraint for that reference which stopped member labels from being createable
- this fixes the schema file and has a migration to drop and recreate the table. Knex handles removal and addition of foreign keys during table drop/create
no issue
* Updated sendEmailWithMagicLink syntax
* Updated label name selection from theme
* Updated migration version for labels
* Added labels to export/import of members
* Added member labels sanitization for case-insensitive duplicates
* Fixed tests
* Fixed label serialization bug on import
* Bumped @tryghost/members-api to 0.15.0
* Fixed lint
* Cleanup
no issue
- The helper allows generating HTML needed to cancel or continue the member's subscription depending on subscription state.
- Added public members endpoint to allow updating subscription's `cancel_at_period_end` attribute available at: `PUT /api/canary/members/subscriptions/:id/`
- Added client-side hook to allow calling subscription cancellation. Allows to create elements with `data-members-cancel-subscription` / `data-members-continue-subscription` attributes which would call subscription update.
- Updated schema and added migration for `current_period_end` column
- As discussed we only add a single column to subscriptions table to avoid preoptimizing for future cases
- Added {{cancel_link}} helper
- Added error handling for {{cancel_link}} when members are disabled
- Added test coverage for {{cancel_link}} helper
- Bumped @tryghost/members-api version to 0.10.2. Needed to use `updateSubscription` middleware
- Bumped gscan to 3.2.0. Needed to recognize new {{cancel_link}} helper
no issue
- Secondary navigation means most nav concepts are supported, e.g. header & footer, or left & right
- The UI is added separately, this PR adds supporting concepts:
- make sure the default value is an empty array
- add support in the API (v3 only)
- add handling in the navigation helper
We want to allow admin users to trigger a retry of failed emails without having to go through the unpublish/republish dance.
- fixed resource identifier in email permissions migration so email permissions are added correctly
- added new email permissions migration so that beta releases can be upgraded without rollback (will be a no-op for any non-beta upgrades)
- added `/emails/:id/retry/` canary Admin API endpoint
- follows same URL pattern as theme activation
- only triggers mega service retry endpoint if the email has a `'failed'` status
no issue
- Increased default mailgun retry limit to 5
- Handling retry logic closer to SDK layer gives less future manual handling
- Allowed failing request to be passed through to the caller
- To be able to handle failed requests more gracefully in the future we need all available error information to be given to the caller
- The previous method with `Promise.all` would have rejected a whole batch without providing details on each specific batch.
- Limited data returned with a failed message to batch values
- Added better error handling on mega layer
- Added new column to store failed batch info
- Added reference to mailgan error docs
- Refactored batch emailer to respond with instances of an object
- It's hard to reason about the response type of bulk mailer when multiple object types can be returned
- This gives more clarity and ability to check with `instanceof` check
no issue
- The limitation on Mailgun side of API seems to be 1000 emails per message.
- The only place where I could find a hard limit of 1000 emails per
batch was this PHP SDK issue: https://github.com/mailgun/mailgun-php/issues/469
- To store ids of sent messages introduce a mega column on the emails table. They can be synced with stats or other metrics during even pooling in the future
- Removed redundant `join(',')` statement.The SDK accepts an array of emails as well. Less code - better code :)
