no-issue
Our base model will only automatically convert numbers to booleans if
the type is 'bool' - however this column was incorrectly added with a
type of 'boolean'. Lucklily - knex with both MySQL & SQLite3 will add
a column with the same type for both of these, so no migration is needed
to fix it.
refs https://github.com/TryGhost/Team/issues/637
The "free" price - when Members signup without using Stripe, should have
a name and description, so that it can be displayed in Portal in a
similar way to paid price's. As there is only ever one, and it is not a
fully fledged price, a setting makes more sense than a dedicated db
table.
refs https://github.com/TryGhost/Team/issues/586
We are no longer using the `stripe_plans` setting, instead we are using
the `stripe_prices` database table. However, we must keep the setting as
the migration from the setting to the database is not done as a standard
migration, but in code. This means our code has to still read and pass
the setting because we will never know if the migration in code has run
yet.
The `portal_plans` setting has been updated to only include 'free' by
default, because the setting must include id's now rather than names.
refs https://github.com/TryGhost/Team/issues/586
A discussion in the Members team resulted in us determining that we do
not need to enforce unique names for Products. Stripe does not enforce
uniqueness for their Products, and we feel it's not necessary for us to.
refs https://github.com/TryGhost/Team/issues/581
refs https://github.com/TryGhost/Team/issues/582
When publishing a post via the API it was possible to send it using `?email_recipient_filter=all/free/paid` which allowed you to send to members only based on their payment status which is quite limiting for some sites.
This PR updates the `?email_recipient_filter` query param to support Ghost's `?filter` param syntax which enables more specific recipient lists, eg:
`?email_recipient_filter=status:free` = free members only
`?email_recipient_filter=status:paid` = paid members only
`?email_recipient_filter=label:vip` = members that have the `vip` label attached
`?email_recipient_filter=status:paid,label:vip` = paid members and members that have the `vip` label attached
The older `free/paid` values are still supported by the API for backwards compatibility.
- updates `Post` and `Email` models to transform legacy `free` and `paid` values to their NQL equivalents on read/write
- lets us not worry about supporting legacy values elsewhere in the code
- cleanup migration to transform all rows slated for 5.0
- removes schema and API `isIn` validations for recipient filters so allow free-form filters
- updates posts API input serializers to transform `free` and `paid` values in the `?email_recipient_filter` param to their NQL equivalents for backwards compatibility
- updates Post API controllers `edit` methods to run a query using the supplied filter to verify that it's valid
- updates `mega` service to use the filter directly when selecting recipients
refs https://github.com/TryGhost/Team/issues/581
Editors are allowed to restrict post visibility and send emails to particular member segments, they need to be able to read labels so that they can select them in a member segment.
refs https://github.com/TryGhost/Team/issues/496
We want to give more control over the default selection of email recipients when publishing a post, to do that we need somewhere to store those settings. These settings are site-wide and intended for use by admins to control the default editor behaviour for all staff users. They _do not_ control API behaviour, if you want to send email when publishing via the API it's still necessary to explicitly opt in to that using the `?email_recipients_filter=` query param.
- new `editor` settings group to indicate that these settings only affect the UI rather than the API
- `editor_default_email_recipients` controls overall behaviour, string/enum with these allowed values:
- `'disabled'`: no option to send email is shown in the editor's publishing dropdown
- `'visibility'`: (default) selected member segment is dynamic and matches the post visibility filter
- `'filter'`: specific member filter defined in `editor_default_email_recipients_filter` setting
- `editor_default_email_recipients_filter` is an NQL string for selecting members, used when `editor_default_email_recipients` is set to `'filter'`
- default value is `'all'`
- the segment string can be any valid NQL filter with the additional special-case values of `'all'` and `'none'`
- single authors were deprecated in v1.22 when we added multiple authors
- we always thought we'd clean this up a lot sooner, but it's stuck because it's an annoying thing to break people's shit over
- still saying "remove in vX" isn't useful, we need to know how long a feature has been deprecated so we can judge whether it's safe to remove
refs https://github.com/TryGhost/Team/issues/586
The `products` and `stripe_prices` tables are missing a description
column which will be used by Portal to display information about the
products and prices
refs 829e8ed010
- i18n is used everywhere but only requires shared or external packages, therefore it's a good candidate for living in shared
- this reduces invalid requires across frontend and server, and lets us use it everywhere until we come up with a better option
- Having these as destructured from the same package is hindering refactoring now
- Events should really only ever be used server-side
- i18n should be a shared module for now so it can be used everywhere until we figure out something better
- Having them seperate also allows us to lint them properly
refs https://github.com/TryGhost/Team/issues/619
As part of the Custom Products work, we are linking members to products
when updating their subscriptions. This requires that we have at least
one product in the database. For existing sites that are using Members
this is handled by the v4.3 03 migration. But for new sites we must
include a fixture.
Also fixes the tests to not reply on the order of the fixtures
refs https://github.com/TryGhost/Team/issues/586
- Adds new `stripe_price_id` column to subscriptions table to store stripe price ids with `index`
- Populates `stripe_price_id` column value to current `plan_id` making the `plan_*` values redundant
- Updates tests
refs https://github.com/TryGhost/Team/issues/586
If a product inside Ghost is deleted, we want to cascade delete all associated Stripe products and prices as they always need to refer back to a ghost product and will hang without any reason otherwise. This change adds cascade delete for products -> stripe_products -> stripe_prices to avoid broken states
refs https://github.com/TryGhost/Team/issues/579
Currently the members signup setting is explicitly yes/no to allowing free members signup, with the implication that when set to "no" members is still active but members have to be created via Stripe or the admin API.
This change renames the setting and changes its type to allow more than a binary option.
- migration to create/update the new setting based on the old value
- free signup = "all", no free signup = "invite"; matches the current UI for this setting
- rename setting everywhere it's used/tested against
- modify `getAllowSelfSignup()` used to configure members packages to only return `true` when the new setting is set to `'all'` to match behaviour to the older setting
- update importer to rename the setting when importing from an older Ghost version
refs https://github.com/TryGhost/Team/issues/618
- The `oauth_client_id` and `oauth_client_secret` are placeholders to store OAuths related data.
- The flag for `oauth_enabled` or anything along those lines was not added intentionally in favour of checking if the `oauth_client_id` & `oauth_client_secret` are null.
refs https://github.com/TryGhost/Team/issues/616
All roles which can publish posts should be able to read/browse products, as content gating
will be based on products going forward.
Creating, updating & destroying products will often make modifications to Stripe which requires
Administrator or Owner roles.
We also improve the permissions tests so that we no longer rely on things being in a particular
order.
refs https://github.com/TryGhost/Team/issues/586
- Change the interval column to be `nullable` as one time payments won't have any interval
- Remove the `livemode` column as we store the connected account's livemode status at top level
no issue
- There is a valid subset of statuses that can be set for the users but there's no "isIn" validation for possible values
- Additionally some of the statuses like warn-1, warn-2, etc. don't have a clear usecase (or at least nothing was found in the codebase for them to be used). They might be up for removal if this assumption is correct
refs https://github.com/TryGhost/Team/issues/586
- Add the `stripe_products` table, so that we can map Stripe Products to Products in Ghost
- Add the `stripe_prices` table, so that we can associate Stripe Prices to Products table
refs https://github.com/TryGhost/Team/issues/586
- Add the products table, so that we can store Products in Ghost
- Add the members_products table, so that we can associate Members w/ Products
- Use sort_order on the members_products table to follow the same convention in members_labels
- Populate the products table with a single product, using the name from the stripe_product_name setting
- Populate the members_products table with relations based on the status column of the members table
Populating the tables allows us to transition from the current system, which does not care about products, into the
new system, where Products are used to group members. The intention is that all existing paid members have the
same product
- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- Fixture posts cover the key features and give users an introduction to how to use their site
- This all comes from the marketing and design teams to refresh the look and feel of Ghost and give users the best possible onboarding experience
Note: this fixture overhaul includes
- new content for new 4.0 features
- regenerated post content using our updated mobiledoc structure
- a switch from British to US English
refs 2bba9989db
- Note: this will require new fixtures so that the navigation links actually work
- These updates are all in aid of getting the best possible default setup and onboarding experinence for new Ghost users
- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- This all comes from the marketing and design teams to refresh the look and feel of Ghost
Note on accent color:
This commit changes the default accent colour again.
The intention is that new sites should get #FF1A75 (pink) as their default.
Any existing sites that do not have an accent colour set yet, should get #15171A (black) on upgrading to 4.0.
These are different as they are different experinces. Fresh sites will be guided to pick a color, so
a bright color is more visible and helps to see what can be done, whilst existing sites get a muted
black, that should be a sensible fall back color.
refs TryGhost/Team#535
We want to ensure that a site will always have a default value of
`'#15171A'` for the accent_color setting.
Since the boot process changed we have three cases to account for:
1. Setting does not exist
2. Setting exists with no value
3. Setting exists with a value
It is only in the case of 2. that we want the migration to update the
database with a default value.
In the case of 3. the site owner has already set a value, which we do
not want to override.
In the case of 1. the setting will be created (and populated with
default value) from the default-settings.json file, by the
populateDefaults method called from the settings service
We also update the accent_color setting to include a non-empty
validation, to ensure that the setting will always have a value, as
sites before 4.x may have an empty accent_color, we must update the
importer to set the default value if one is not present. Otherwise we
would run into validation errors and even if we didn't would have an
invalid database state.
no-issue
We were originally checking the state of the database, e.g. if a foreign
key constraint existed, so that we could conditionally act upon it. This
was to ensure that our migrations are idempotent.
Some database configurations, for example if you have many databases on
a single MySQL instance, would cause these information_schema queries to
take an exceptionally long time.
In order to speed up migrations, we instead attempt the action we want
to apply to the database, and then catch relevant errors to ensure the
migration is idempotent.
SQLite does not error when adding duplicate foreign or primary key
constraints, meaning that we must keep in pre-checks for these
operations, when running on SQLite
Co-authored-by: Daniel Lockyer <hi@daniellockyer.com>
refs: https://github.com/TryGhost/Team/issues/510
- In the case that host config is provided, keep staff users within the limiti
- The definition of a staff user is a user with a role other than Contributor, and whose status is not inactive
- Contributors don't count
- Suspended (status inactive) users don't count
- Locked users DO count
- Invited users DO count
- You can't invite more staff users whilst there are pending invites
- You can't unsuspend a user, or change the role on a user in such a way as will take you over your limit
- You can't import staff users - all imported users are automatically set to Contributors
- As part of this work, we are changing the default Ghost user to a Contributor otherwise it uses up a staff user
Note: there is one known active bug with this commit.
- Assume you have one remaining user within your limit. You send an invite, this works.
- You cannot "resend" that invite, it will think you're sending a new invite and hit the limit
- You must "revoke" that invite first, and create a new one
- This bug exists because the resend function uses the add endpoint & does a delete+add, but this hits the permission check before the delete
refs https://github.com/TryGhost/Team/issues/513
refs https://github.com/TryGhost/Team/issues/477
- We have skipped work to improve the 'defaultTo' value when working on Ghost 4.0 release, so adding this comment while context loaded
- defaultTo should not be set to anything as it leads to more maintenance work during major version bump
- having validation might make sense but could lead to similar maintenance work unless it's linked to some global nosion of "supported API versions" used everywhere
refs https://github.com/TryGhost/Ghost/pull/12598
- This changeset adds idepmotence to situations where unique contraint has to be dropped or added to the table
- Note '4.0/07-alter-unique-constraint-for-posts-slug.js` was migration that was effected by lack of idempotence
refs https://github.com/TryGhost/Ghost/issues/10318
- Storing JSON object in settings has caused multiple bugs in the past and was considered an antipattern
- This is a last bit in the long process of getting rid of "object" settings in the database. At this point there should be no settings with this type. Yey!
no issue
- all migrations should contain a logging message
- info for successful things, or warn if we deviated from what was
expected for some reason
- also added some spacing to make them easier to read
refs https://github.com/TryGhost/Team/issues/332
- The last value that has been used in the code was "members"
- By default members will be always "on" starting Ghost 4.0, so there's no need for this flag anymore
- Therefore there's no real need to keep "labs" around
refs https://github.com/TryGhost/Ghost/issues/12602
* Updated members_status_events table
By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range
* Updated MEGA to handle addition of 'comped' status
With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"
* Updated customQuery for MemberStatusEvent
With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.
* Populated members_status_events table
As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.
The assumptions used in backfilling is that a Member's current status,
is their only status.
refs https://github.com/TryGhost/Ghost/issues/10318
- Object format used in previous "slack" setting was considered an
anti-pattern. Flag structure of separate slack_url and slack_username
values was extracted out of the "slack" JSON.
refs https://github.com/TryGhost/Ghost/issues/12602
As we only want to add events for when an email is changed, we need to store the previous email if we want a complete log of all emails for a member
refs https://github.com/TryGhost/Ghost/issues/12602
* Added members_payment_events table
This table will store successful and unsuccessful payment attempts, and
can be used to calculate gross volume over time.
* Added members_login_events table
This table can be used to audit member logins
* Added members_email_change_events table
This table will allow us to store a history of email addresses associated with a member
* Added members_status_events table
This table will allow us to track the change in status over time for members, as well
as calculate aggregates over time, e.g. paid members over time
* Added members_paid_subscription_events
This table will allow us to track subscriptions changes for members, as well as
calculating MRR over time
refs https://github.com/TryGhost/Ghost/issues/12567
- Changing unique constraint from slug to slug+type should allow for posts and pages to be created with the same slug
- The constraint will be present on application layer for API v4 while we figure out how to deal with it in API v5
refs https://github.com/TryGhost/Ghost/issues/12567
- The method was using unneeded lodash dependency and was too complex for what it was doing
- Reshuffled internal code to use native JS filter/forEach iterators
refs https://github.com/TryGhost/Ghost/issues/12567
- Introduced here @@UNIQUE_CONSTRAINTS@@ notation allows to create unique contraints over multiple database fields. This will be needed to change posts' table unique constraint from `slug` to `slug+type`.
- The notation is equivalent to SQL's: UNIQUE(column_name1,column_name2)
- Example use in schema:
posts: {
slug: {type: 'string', maxlength: 191, nullable: false},
type: {type: 'string', maxlength: 50, nullable: false, defaultTo: 'post', validations: {isIn: [['post', 'page']]}},
'@@UNIQUES@@': [
['slug', 'type']
]
}
no-issue
This value is nullable from the Stripe API so we're making sure that we
can store exact values locally.
SQLite3 does not supports altering columns so instead we have to:
1. Create a temp table and copy the data to it
2. Delete the original table
3. Recreate the original table with the necessary modifications
4. Copy the data from the temp table
5. Drop the temp table
refs #12160
This flag will allow us easier filtering of members via the API
* Added status column to members table
This flag will be used to determine if a member is free or paid, rather
than relying on joins with the customers and subscriptions tables.
* Added migration to populate members.status
As we add the column with a default value of "free" we only need to care
about the paid members here. We also preemptively handle migrations for
SQLite where there are > 998 paid members.
no refs
Adds new FirstPromoter settings similar to amp, which allows sites to take advantage of FirstPromoter to launch their own member referral program natively.
- Adds new firstpromoter settings group
- Adds `firstpromoter` setting to group
- Adds `firstpromoter_id` setting to group for FirstPromoter referral tracking id
- Updated tests
closes#12181
refs https://github.com/TryGhost/Ghost/pull/12265
- The 'core' value is invalid for settings 'type' column
- The 'core' default value for 'type' column in model is also invalid
- Both need to be removed as they are never used and only introduce confusion into the codebase
refs https://github.com/TryGhost/Ghost/issues/12461
- adds `members.email_count` and `members.email_opened_count` columns to contain cached counts for faster queries when outputting member data via API
- adds migration to populate cached counts with existing data
- tested locally on ~50k members which took ~4sec on mysql
- updates members output serializer to include the new fields in API output
refs https://github.com/TryGhost/Ghost/issues/12421
- nullable so we can distinguish between members that have and haven't received any trackable emails
- indexed because we'll be using this column for sorting
no issue
- cleans up unused tables `emails.{meta,stats}`
- adds timestamp columns `email_recipients.{delivered_at,opened_at,failed_at}` that can be used for event timelines and basic stats aggregation
- indexed because we want to sort by these columns to find the "latest event" when limiting Mailgun events API requests
- adds aggregated stats columns `emails.{delivered_count,opened_count,failed_count}`
- adds a composite index on `email_recipients.[email_id,member_email]` to dramatically speed up `email_recipient` update queries when processing events
- modifies the db initialisation to support an `'@@INDEXES@@'` key in table schema definition for composite indexes
- these helpers remove a lot of the duplicated code that we had when doing up/down column migrations, and provides a much shorter way of doing this in the future
no issue
Updated newsletter badge to be shown by default which was previously set to be hidden. Users can still control the visibility via Email settings in Ghost-Admin
no-issue
We want to give users to ability to customise the content of their newsletter, and the first step
toward that is a setting in which we can store text or html to embed in the template
no-issue
* Handled send_email_when_published in Posts API
This restores backwards compatibility of the Posts API allowing existing
clients to continue to use the `send_email_when_published` flag. This
change uses two edits, which is unfortunate. The reason being is that
this is an API compatibility issue, not a model issue, so we shouldn't
introduce code to the model layer to handle it. The visibility property
of the model is used to determine how to fall back, and because it can
be left out of the API request, and relies on a default in the settings,
we require that the model decide on the `visibility` before we run our
fallback logic (or we duplicate the `visibility` default at the cost of
maintenance in the future)
* Dropped send_email_when_published column from posts
Since this column is not used any more, we can drop it from the table.
We include an extra migration to repopulate the column in the event of
a rollback
* Updated importer to handle send_email_when_published
Because we currently export this value from Ghost, we should correctly
import it. This follows the same logic as the migrations for this value.
* Included send_email_when_published in API response
As our v3 API documentation includes `send_email_when_published` we must
retain backward compatibility by calculating the property.
* Fixed fields filter with send_email_when_published
* Added safety checks to frame properties
Some parts of the code pass a manually created "frame" which is missing
lots of properties, so we check for the existence of all of them before
using them.
* Fixed 3.1 migration to include columnDefinition
We require that migrations have all the information they need contained
within them as they run in an unknown state of the codebase, which could
be from the commit they are introduced, to any future commit. In this
case the column definition is removed from the schema in 3.38 and the
migration would fail when run in this version or later.
refs https://github.com/TryGhost/Ghost/issues/12355
- Adds new default settings for newsletter customisations - `newsletter_show_badge`, `newsletter_show_header` and `newsletter_body_font_category`
- Adds migrations to update group for new settings
- Add migration to update settings based on existing config value for newsletter settings
- Passes new newsletter settings to newsletter template and updates design based on them
- Fix tests
no issue
- tracking of bulk email opens can be enabled/disabled over time, if we're calculating analytics for emails we don't want emails which didn't have tracking enabled skewing the results so we need a record of whether tracking was enabled for each email
no-issue
This column will allow us to store the canonical recipient filter on the
email resource giving us a detailed log of which members an email was
intended for
no-issue
This column will allow us to decouple the recipients of newsletter from
the `visibility` of a post, allowing us to send emails to specifically
free members, or to send paid posts as newsletters to all members.
no issue
- all staff users can browse/read snippets so they're usable in the editor for everyone
- only administrators, editors, and admin integrations are able to create/edit/delete snippets
no issue
- minimal table structure required for the first iteration of content snippets
- snippets are stored pieces of re-usable content that could effectively be entire posts so the `mobiledoc` field length matches the `posts.mobiledoc` field length
no issue
The email table should be a reference for all data that was used when sending an email. From and Reply-to addresses can change over time and we don't have any other reference for their value at the time of sending an email so we should store them alongside the email content.
- schema updated with `from` and `reply_to` columns
- both are set to `nullable` because we don't have historic data (can be populated and changed in later migrations if needed)
- neither `from` or `reply_to` have `isEmail` validations because they can have name+email in an email-specific format
- will help keep concerns separated in the future. `mega` service can deal with all of the email contents/properties, and the `bulk-email` service's concerns are then only email sending and any provider-specific needs
no-issue
After discussion with Matt, we decided that 192 bits for the token is a
good number, as it has no padding when base64 encoded and is more secure
than 128 bits, whilst still a managable size.
no-issue
This is a table to store single use tokens for use in magic links, the
columns are as simple as possible at the moment and are designed as:
id - standard ObjectID like all of our tables
token - 128bit base64 encoded string
data - arbitrary data to store against the token
created_at - timestamp to allow for expiry to be implemented for tokens
no issue
We want to store a list of recipients for each bulk email so that we have a consistent set of data that background processing/sending jobs can work from without worrying about moving large data sets around or member data changing mid-send.
- `email_batches` table acts as a join table with status for email<->email_recipient
- stores a provider-specific ID that we get back when submitting a batch for sending to the bulk email provider
- `status` allows for batch-specific status updates and picking up where we left off when submitting batches if needed
- explicitly tying a list of email recipients to a batch allows for partial retries
- `email_recipients` table acts as a join table for email<->member
- `member_id` does not have a foreign key constraint because members can be deleted but does have an index so that we can efficiently query which emails a member has received
- stores static copies of the member info present at the time of sending an email for consistency in background jobs and auditing/historical data
refs #11999
- The `routes_hash` setting will be used during the boot process to update the hash
of currently loaded routes.yaml file in case it's different from last restart
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account, public setting
- `members_reply_address` - Where you receive responses to newsletters
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account
- `members_reply_address` - Where you receive responses to newsletters
refs #12126
- Adds migration to add impersonation permission to administrators
- Adds default permission fixture to allow administrators to read member impersonation urls
- Allows administrators to create member impersonation magic links
- deleted files under `core/server/lib/promise` and related test files
- added `@tryghost/promise` as a dependency
- fixed all local requires to point to the new package
no-issue
We are in the process of creating migrations to add foreign key constraints
and cascading deletes to the members_stripe_* tables to make listing members
and deleting members faster. As well as the migrations we need to update the
database schema so that new installations have the correct indexes and constraints.
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
no-issue
They will be used to store webhook information so that we can persist it between
boots and simplify the creation process of webhooks in members
no issue
- Adds new portal settings - `portal_button_style`, `portal_button_icon` and `portal_button_signup_text`
- New settings allows customization of portal button
- Updates tests to include new settings
