refs https://github.com/TryGhost/Team/issues/710
refs https://github.com/TryGhost/Team/issues/725
Products will now have a single monthly and yearly price which will be
used throughout Themes, Portal & Admin. These columns will be used to
track the current prices for each of them, and will update anytime we
change the pricing of a product.
Due to a circular table dependency we have not added a foreign key
constraint to the new columns, this will be handled at a later date. It
is tracked in issue 725 references above
refs https://github.com/TryGhost/Team/issues/693
Since we've got rid of the concept of Complimentary with the Custom
Prices work, we're removing the 'comped' status from members. This
involves a migration for existing members, a schema update for the
validation, and a bump to members-api to no longer use the 'comped'
status for new members.
We also update the aggregation of the MemberStatusEvent to consider the
'comped' status as 'paid', and that there are 0 'comped' status events
in the database.
We can consider a migration for this data in the future, either adding
new status events moving from 'comped' to 'paid', or by modifying
existing status events. However both of these are very difficulty to
write a down migration for, and might be best saved for a major version.
- @tryghost/members-api@1.7.0 is the version that includes the required
changes, however we have already bumped to 1.8.0 in Ghost
refs https://github.com/TryGhost/Team/issues/581
closes https://github.com/TryGhost/Team/issues/582
Emails can now be sent to members with specific associated labels or products by specifying an NQL string. We want to bring the same members segment feature to content by allowing `visibility` to be an NQL filter string on top of the `public/members/paid` special-case strings.
As an example it's possible to set `posts.visibility` to `label:vip` to make a post available only to those members with the `vip` label.
- removed enum validations for `visibility` so it now accepts any string or `null`
- bumped `@tryghost/admin-api-schema` for API-level validation changes
- added nql validation to API input validators by running the visibility query against the members model
- added transform of NQL to special-case visibility values when saving post model
- ensures there's a single way of representing "members" and "paid" where NQL gives multiple ways of representing the same segment
- useful for keeping theme-level checks such as `{{#has visibility="paid"}}` working as expected
- updated content-gating to parse nql from post's visibility and use it to query the currently logged in member to see if there's a match
- bumped @tryghost/members-api to include label and product data when loading member
no-issue
Our base model will only automatically convert numbers to booleans if
the type is 'bool' - however this column was incorrectly added with a
type of 'boolean'. Lucklily - knex with both MySQL & SQLite3 will add
a column with the same type for both of these, so no migration is needed
to fix it.
refs https://github.com/TryGhost/Team/issues/586
A discussion in the Members team resulted in us determining that we do
not need to enforce unique names for Products. Stripe does not enforce
uniqueness for their Products, and we feel it's not necessary for us to.
refs https://github.com/TryGhost/Team/issues/581
refs https://github.com/TryGhost/Team/issues/582
When publishing a post via the API it was possible to send it using `?email_recipient_filter=all/free/paid` which allowed you to send to members only based on their payment status which is quite limiting for some sites.
This PR updates the `?email_recipient_filter` query param to support Ghost's `?filter` param syntax which enables more specific recipient lists, eg:
`?email_recipient_filter=status:free` = free members only
`?email_recipient_filter=status:paid` = paid members only
`?email_recipient_filter=label:vip` = members that have the `vip` label attached
`?email_recipient_filter=status:paid,label:vip` = paid members and members that have the `vip` label attached
The older `free/paid` values are still supported by the API for backwards compatibility.
- updates `Post` and `Email` models to transform legacy `free` and `paid` values to their NQL equivalents on read/write
- lets us not worry about supporting legacy values elsewhere in the code
- cleanup migration to transform all rows slated for 5.0
- removes schema and API `isIn` validations for recipient filters so allow free-form filters
- updates posts API input serializers to transform `free` and `paid` values in the `?email_recipient_filter` param to their NQL equivalents for backwards compatibility
- updates Post API controllers `edit` methods to run a query using the supplied filter to verify that it's valid
- updates `mega` service to use the filter directly when selecting recipients
- single authors were deprecated in v1.22 when we added multiple authors
- we always thought we'd clean this up a lot sooner, but it's stuck because it's an annoying thing to break people's shit over
- still saying "remove in vX" isn't useful, we need to know how long a feature has been deprecated so we can judge whether it's safe to remove
refs https://github.com/TryGhost/Team/issues/586
The `products` and `stripe_prices` tables are missing a description
column which will be used by Portal to display information about the
products and prices
refs https://github.com/TryGhost/Team/issues/586
- Adds new `stripe_price_id` column to subscriptions table to store stripe price ids with `index`
- Populates `stripe_price_id` column value to current `plan_id` making the `plan_*` values redundant
- Updates tests
refs https://github.com/TryGhost/Team/issues/586
If a product inside Ghost is deleted, we want to cascade delete all associated Stripe products and prices as they always need to refer back to a ghost product and will hang without any reason otherwise. This change adds cascade delete for products -> stripe_products -> stripe_prices to avoid broken states
refs https://github.com/TryGhost/Team/issues/586
- Change the interval column to be `nullable` as one time payments won't have any interval
- Remove the `livemode` column as we store the connected account's livemode status at top level
no issue
- There is a valid subset of statuses that can be set for the users but there's no "isIn" validation for possible values
- Additionally some of the statuses like warn-1, warn-2, etc. don't have a clear usecase (or at least nothing was found in the codebase for them to be used). They might be up for removal if this assumption is correct
refs https://github.com/TryGhost/Team/issues/586
- Add the `stripe_products` table, so that we can map Stripe Products to Products in Ghost
- Add the `stripe_prices` table, so that we can associate Stripe Prices to Products table
refs https://github.com/TryGhost/Team/issues/586
- Add the products table, so that we can store Products in Ghost
- Add the members_products table, so that we can associate Members w/ Products
- Use sort_order on the members_products table to follow the same convention in members_labels
- Populate the products table with a single product, using the name from the stripe_product_name setting
- Populate the members_products table with relations based on the status column of the members table
Populating the tables allows us to transition from the current system, which does not care about products, into the
new system, where Products are used to group members. The intention is that all existing paid members have the
same product
refs https://github.com/TryGhost/Team/issues/513
refs https://github.com/TryGhost/Team/issues/477
- We have skipped work to improve the 'defaultTo' value when working on Ghost 4.0 release, so adding this comment while context loaded
- defaultTo should not be set to anything as it leads to more maintenance work during major version bump
- having validation might make sense but could lead to similar maintenance work unless it's linked to some global nosion of "supported API versions" used everywhere
refs https://github.com/TryGhost/Ghost/issues/10318
- Storing JSON object in settings has caused multiple bugs in the past and was considered an antipattern
- This is a last bit in the long process of getting rid of "object" settings in the database. At this point there should be no settings with this type. Yey!
refs https://github.com/TryGhost/Ghost/issues/12602
* Updated members_status_events table
By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range
* Updated MEGA to handle addition of 'comped' status
With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"
* Updated customQuery for MemberStatusEvent
With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.
* Populated members_status_events table
As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.
The assumptions used in backfilling is that a Member's current status,
is their only status.
refs https://github.com/TryGhost/Ghost/issues/12602
As we only want to add events for when an email is changed, we need to store the previous email if we want a complete log of all emails for a member
refs https://github.com/TryGhost/Ghost/issues/12602
* Added members_payment_events table
This table will store successful and unsuccessful payment attempts, and
can be used to calculate gross volume over time.
* Added members_login_events table
This table can be used to audit member logins
* Added members_email_change_events table
This table will allow us to store a history of email addresses associated with a member
* Added members_status_events table
This table will allow us to track the change in status over time for members, as well
as calculate aggregates over time, e.g. paid members over time
* Added members_paid_subscription_events
This table will allow us to track subscriptions changes for members, as well as
calculating MRR over time
refs https://github.com/TryGhost/Ghost/issues/12567
- Changing unique constraint from slug to slug+type should allow for posts and pages to be created with the same slug
- The constraint will be present on application layer for API v4 while we figure out how to deal with it in API v5
no-issue
This value is nullable from the Stripe API so we're making sure that we
can store exact values locally.
SQLite3 does not supports altering columns so instead we have to:
1. Create a temp table and copy the data to it
2. Delete the original table
3. Recreate the original table with the necessary modifications
4. Copy the data from the temp table
5. Drop the temp table
refs #12160
This flag will allow us easier filtering of members via the API
* Added status column to members table
This flag will be used to determine if a member is free or paid, rather
than relying on joins with the customers and subscriptions tables.
* Added migration to populate members.status
As we add the column with a default value of "free" we only need to care
about the paid members here. We also preemptively handle migrations for
SQLite where there are > 998 paid members.
closes#12181
refs https://github.com/TryGhost/Ghost/pull/12265
- The 'core' value is invalid for settings 'type' column
- The 'core' default value for 'type' column in model is also invalid
- Both need to be removed as they are never used and only introduce confusion into the codebase
refs https://github.com/TryGhost/Ghost/issues/12461
- adds `members.email_count` and `members.email_opened_count` columns to contain cached counts for faster queries when outputting member data via API
- adds migration to populate cached counts with existing data
- tested locally on ~50k members which took ~4sec on mysql
- updates members output serializer to include the new fields in API output
refs https://github.com/TryGhost/Ghost/issues/12421
- nullable so we can distinguish between members that have and haven't received any trackable emails
- indexed because we'll be using this column for sorting
no issue
- cleans up unused tables `emails.{meta,stats}`
- adds timestamp columns `email_recipients.{delivered_at,opened_at,failed_at}` that can be used for event timelines and basic stats aggregation
- indexed because we want to sort by these columns to find the "latest event" when limiting Mailgun events API requests
- adds aggregated stats columns `emails.{delivered_count,opened_count,failed_count}`
- adds a composite index on `email_recipients.[email_id,member_email]` to dramatically speed up `email_recipient` update queries when processing events
- modifies the db initialisation to support an `'@@INDEXES@@'` key in table schema definition for composite indexes
no-issue
* Handled send_email_when_published in Posts API
This restores backwards compatibility of the Posts API allowing existing
clients to continue to use the `send_email_when_published` flag. This
change uses two edits, which is unfortunate. The reason being is that
this is an API compatibility issue, not a model issue, so we shouldn't
introduce code to the model layer to handle it. The visibility property
of the model is used to determine how to fall back, and because it can
be left out of the API request, and relies on a default in the settings,
we require that the model decide on the `visibility` before we run our
fallback logic (or we duplicate the `visibility` default at the cost of
maintenance in the future)
* Dropped send_email_when_published column from posts
Since this column is not used any more, we can drop it from the table.
We include an extra migration to repopulate the column in the event of
a rollback
* Updated importer to handle send_email_when_published
Because we currently export this value from Ghost, we should correctly
import it. This follows the same logic as the migrations for this value.
* Included send_email_when_published in API response
As our v3 API documentation includes `send_email_when_published` we must
retain backward compatibility by calculating the property.
* Fixed fields filter with send_email_when_published
* Added safety checks to frame properties
Some parts of the code pass a manually created "frame" which is missing
lots of properties, so we check for the existence of all of them before
using them.
* Fixed 3.1 migration to include columnDefinition
We require that migrations have all the information they need contained
within them as they run in an unknown state of the codebase, which could
be from the commit they are introduced, to any future commit. In this
case the column definition is removed from the schema in 3.38 and the
migration would fail when run in this version or later.
no issue
- tracking of bulk email opens can be enabled/disabled over time, if we're calculating analytics for emails we don't want emails which didn't have tracking enabled skewing the results so we need a record of whether tracking was enabled for each email
no-issue
This column will allow us to store the canonical recipient filter on the
email resource giving us a detailed log of which members an email was
intended for
no-issue
This column will allow us to decouple the recipients of newsletter from
the `visibility` of a post, allowing us to send emails to specifically
free members, or to send paid posts as newsletters to all members.
no issue
- minimal table structure required for the first iteration of content snippets
- snippets are stored pieces of re-usable content that could effectively be entire posts so the `mobiledoc` field length matches the `posts.mobiledoc` field length
no issue
The email table should be a reference for all data that was used when sending an email. From and Reply-to addresses can change over time and we don't have any other reference for their value at the time of sending an email so we should store them alongside the email content.
- schema updated with `from` and `reply_to` columns
- both are set to `nullable` because we don't have historic data (can be populated and changed in later migrations if needed)
- neither `from` or `reply_to` have `isEmail` validations because they can have name+email in an email-specific format
- will help keep concerns separated in the future. `mega` service can deal with all of the email contents/properties, and the `bulk-email` service's concerns are then only email sending and any provider-specific needs
no-issue
After discussion with Matt, we decided that 192 bits for the token is a
good number, as it has no padding when base64 encoded and is more secure
than 128 bits, whilst still a managable size.
no-issue
This is a table to store single use tokens for use in magic links, the
columns are as simple as possible at the moment and are designed as:
id - standard ObjectID like all of our tables
token - 128bit base64 encoded string
data - arbitrary data to store against the token
created_at - timestamp to allow for expiry to be implemented for tokens
no issue
We want to store a list of recipients for each bulk email so that we have a consistent set of data that background processing/sending jobs can work from without worrying about moving large data sets around or member data changing mid-send.
- `email_batches` table acts as a join table with status for email<->email_recipient
- stores a provider-specific ID that we get back when submitting a batch for sending to the bulk email provider
- `status` allows for batch-specific status updates and picking up where we left off when submitting batches if needed
- explicitly tying a list of email recipients to a batch allows for partial retries
- `email_recipients` table acts as a join table for email<->member
- `member_id` does not have a foreign key constraint because members can be deleted but does have an index so that we can efficiently query which emails a member has received
- stores static copies of the member info present at the time of sending an email for consistency in background jobs and auditing/historical data
no-issue
We are in the process of creating migrations to add foreign key constraints
and cascading deletes to the members_stripe_* tables to make listing members
and deleting members faster. As well as the migrations we need to update the
database schema so that new installations have the correct indexes and constraints.
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates default settings to contain correct type and validation for each setting
- Updates `populateDefaults` to correctly insert type value for new settings
- Updates settings schema to allow only select types - `array`, `number`, `boolean`, `string`
- `object` is a temporary type allowed till we get rid of all JSON object settings
