refs 829e8ed010
- i18n is used everywhere but only requires shared or external packages, therefore it's a good candidate for living in shared
- this reduces invalid requires across frontend and server, and lets us use it everywhere until we come up with a better option
refs https://github.com/TryGhost/Team/issues/579
The previous `members_allow_free_signup` -> `members_signup_access` migration made a direct correlation between the toggle `true/false` to `all/invite` under the assumption that behaviour between the two settings would be identical. The assumption was incorrect and the behaviour is changing so `invite` forces invite-only mode, stopping all front-end signup to free or paid plans with the free plan now being disabled via the portal plans setting.
- check existing `members_signup_access` setting and if it's `'invite'` migrate it to `'all'` where signup should still be possible. The "invite-only" mode should only be active if certain conditions are met:
- Stripe is not configured ("allow free member signup" off and no Stripe showed "invite-only" in portal)
- Stripe is configured but no plans are selected in portal (no plans showed "invite-only" in portal)
- when migrating `'invite'` to `'all'`, also remove `'free'` plan from the `portal_plans` setting to avoid previously paid-only sites unexpectedly showing a free plan on signup
refs https://github.com/TryGhost/Team/issues/634
- find earliest backup file created when a 4.3 migration was run, if found use the `members_allow_free_signup` value from there to change `members_signup_access` from `'all'` to `'invite'` if necessary
- Having these as destructured from the same package is hindering refactoring now
- Events should really only ever be used server-side
- i18n should be a shared module for now so it can be used everywhere until we figure out something better
- Having them seperate also allows us to lint them properly
refs https://github.com/TryGhost/Team/issues/634
- the migration moving `members_allow_free_signup` to `members_signup_access` was expecting a raw boolean setting value but the actual value is a string so always evaluated as truthy making all sites look like they had "allow free members signup" toggled on when generating the new setting's value
- updated to check for an explicit string value in `up` and set an explicit string value in `down`
refs c873899e49
- as of `bson-objectid` v2.0.0, this library exports the function
to generate an ObjectID directly, and then you need to use `.toHexString()`
to get the 24 character hex string - 6696f27d82
- this commit removes all uses of `.generate()` and replaces with this
change
refs https://github.com/TryGhost/Team/issues/619
As part of the Custom Products work, we are linking members to products
when updating their subscriptions. This requires that we have at least
one product in the database. For existing sites that are using Members
this is handled by the v4.3 03 migration. But for new sites we must
include a fixture.
Also fixes the tests to not reply on the order of the fixtures
refs https://github.com/TryGhost/Team/issues/586
- Adds new `stripe_price_id` column to subscriptions table to store stripe price ids with `index`
- Populates `stripe_price_id` column value to current `plan_id` making the `plan_*` values redundant
- Updates tests
refs https://github.com/TryGhost/Team/issues/586
If a product inside Ghost is deleted, we want to cascade delete all associated Stripe products and prices as they always need to refer back to a ghost product and will hang without any reason otherwise. This change adds cascade delete for products -> stripe_products -> stripe_prices to avoid broken states
refs https://github.com/TryGhost/Team/issues/579
Currently the members signup setting is explicitly yes/no to allowing free members signup, with the implication that when set to "no" members is still active but members have to be created via Stripe or the admin API.
This change renames the setting and changes its type to allow more than a binary option.
- migration to create/update the new setting based on the old value
- free signup = "all", no free signup = "invite"; matches the current UI for this setting
- rename setting everywhere it's used/tested against
- modify `getAllowSelfSignup()` used to configure members packages to only return `true` when the new setting is set to `'all'` to match behaviour to the older setting
- update importer to rename the setting when importing from an older Ghost version
refs https://github.com/TryGhost/Team/issues/618
- The `oauth_client_id` and `oauth_client_secret` are placeholders to store OAuths related data.
- The flag for `oauth_enabled` or anything along those lines was not added intentionally in favour of checking if the `oauth_client_id` & `oauth_client_secret` are null.
refs https://github.com/TryGhost/Team/issues/616
All roles which can publish posts should be able to read/browse products, as content gating
will be based on products going forward.
Creating, updating & destroying products will often make modifications to Stripe which requires
Administrator or Owner roles.
We also improve the permissions tests so that we no longer rely on things being in a particular
order.
refs https://github.com/TryGhost/Team/issues/586
- Change the interval column to be `nullable` as one time payments won't have any interval
- Remove the `livemode` column as we store the connected account's livemode status at top level
no issue
- There is a valid subset of statuses that can be set for the users but there's no "isIn" validation for possible values
- Additionally some of the statuses like warn-1, warn-2, etc. don't have a clear usecase (or at least nothing was found in the codebase for them to be used). They might be up for removal if this assumption is correct
refs https://github.com/TryGhost/Team/issues/586
- Add the `stripe_products` table, so that we can map Stripe Products to Products in Ghost
- Add the `stripe_prices` table, so that we can associate Stripe Prices to Products table
refs https://github.com/TryGhost/Team/issues/586
- Add the products table, so that we can store Products in Ghost
- Add the members_products table, so that we can associate Members w/ Products
- Use sort_order on the members_products table to follow the same convention in members_labels
- Populate the products table with a single product, using the name from the stripe_product_name setting
- Populate the members_products table with relations based on the status column of the members table
Populating the tables allows us to transition from the current system, which does not care about products, into the
new system, where Products are used to group members. The intention is that all existing paid members have the
same product
closes https://github.com/TryGhost/Team/issues/595
Due to a bug in `mrr_delta` calculation, we ended up reducing the MRR delta by twice the original amount when a subscription goes from active to canceled and storing it in `members_paid_subscription_events` table, which is used to show the MRR chart on Dashboard. The way we identify the incorrect events in the table which got the double negative value is by checking if they match certain criteria - Both `from_plan` and `to_plan` have same value as a subscription changes status while being on same plan.
This migration halves the `mrr_delta` for incorrect events to restore the correct MRR change for the site.
refs https://github.com/TryGhost/Team/issues/555
- Export files included a lot of data which was not used in the importer, for example: members, labels, migrations and many more. This lead to a lot of clutter in the import files and made it hard to reason about their purpose.
- The main purpose of exports - is to export importable resources. These are posts, tags, and users. The rest of data like members or migrations either have their own importer (like CSV importer for members) or does not and should not have any ways to be imported.
- These changes are in now way complete. It's a first step towards resource-based exports which could be properly versioned in the future on API level and not be a mirror of the DB structure.
- This is sort of a breaking change. But we are doing it because: (1) its an internal API that should not be used by external clients, (2) there was no public contract to have this API stable at any point, (3) we really need to get back the control over export files structure and size
- In case an external client was dependent on some structure of the exported json file they can still pass in ALL of previously exported data by passing table names in `include` query parameter.
refs https://github.com/TryGhost/Team/issues/555
- Previous blocklist approach was resulting in adding every single new table into an export automatically. Which creates possibility to leak sensitive data if not used porperly. Allowlist approach gives better control over what is exported, makes this information explicit, and version-control friendlier
refs https://github.com/TryGhost/Team/issues/555
- The getVersionAndTables was doing too much and was only used once creating clutter in doExport method. Refactored code doing 2 direct calls instead of destructuring "dances".
closes https://github.com/TryGhost/Team/issues/557
After updating to v4 on SQLite from v3, we were unable to delete members due to a foreign key mismatch error. This is because the migrations which recreates the tables for `members_stripe_customers_subscriptions` and `members_stripe_customers` doesn't add back the unique constraint which is needed by FKs for reference.
The migration creates the missing UNIQUE constraints on the tables.
- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- Fixture posts cover the key features and give users an introduction to how to use their site
- This all comes from the marketing and design teams to refresh the look and feel of Ghost and give users the best possible onboarding experience
Note: this fixture overhaul includes
- new content for new 4.0 features
- regenerated post content using our updated mobiledoc structure
- a switch from British to US English
refs 2bba9989db
- Note: this will require new fixtures so that the navigation links actually work
- These updates are all in aid of getting the best possible default setup and onboarding experinence for new Ghost users
- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- This all comes from the marketing and design teams to refresh the look and feel of Ghost
Note on accent color:
This commit changes the default accent colour again.
The intention is that new sites should get #FF1A75 (pink) as their default.
Any existing sites that do not have an accent colour set yet, should get #15171A (black) on upgrading to 4.0.
These are different as they are different experinces. Fresh sites will be guided to pick a color, so
a bright color is more visible and helps to see what can be done, whilst existing sites get a muted
black, that should be a sensible fall back color.
refs TryGhost/Team#535
We want to ensure that a site will always have a default value of
`'#15171A'` for the accent_color setting.
Since the boot process changed we have three cases to account for:
1. Setting does not exist
2. Setting exists with no value
3. Setting exists with a value
It is only in the case of 2. that we want the migration to update the
database with a default value.
In the case of 3. the site owner has already set a value, which we do
not want to override.
In the case of 1. the setting will be created (and populated with
default value) from the default-settings.json file, by the
populateDefaults method called from the settings service
We also update the accent_color setting to include a non-empty
validation, to ensure that the setting will always have a value, as
sites before 4.x may have an empty accent_color, we must update the
importer to set the default value if one is not present. Otherwise we
would run into validation errors and even if we didn't would have an
invalid database state.
no-issue
We were originally checking the state of the database, e.g. if a foreign
key constraint existed, so that we could conditionally act upon it. This
was to ensure that our migrations are idempotent.
Some database configurations, for example if you have many databases on
a single MySQL instance, would cause these information_schema queries to
take an exceptionally long time.
In order to speed up migrations, we instead attempt the action we want
to apply to the database, and then catch relevant errors to ensure the
migration is idempotent.
SQLite does not error when adding duplicate foreign or primary key
constraints, meaning that we must keep in pre-checks for these
operations, when running on SQLite
Co-authored-by: Daniel Lockyer <hi@daniellockyer.com>
no issue
Upgrading from v1 or v2 can result in successful upgrades but with mobiledoc errors showing in the logs:
```
NAME: InternalServerError
MESSAGE: Mobiledoc card 'card-markdown' not found.
```
The errors do not signify a problem as long as the 4.0 migrations run because those rename the deprecated card before re-rendering.
- `@tryghost/kg-default-cards` dropped support for `card-markdown` cards. 4.0 migrations handled this by renaming all `card-markdown` cards to `markdown` before re-generating any content
- 2.0 and 3.0 also had migrations that re-generated content but they are run before the 4.0 card rename migration meaning that the mobiledoc renderer sees cards that it doesn't know about. The behaviour for unknown cards is to log an error and skip rendering of that card
- by NOOPing the 2.0 and 3.0 migrations we eliminate the incompatibility errors and reduce the amount of processing the upgrade needs to perform
closes https://github.com/TryGhost/Team/issues/467
- switches to storing "transform-ready" URLs in the database
- transform-ready URLs contain a `__GHOST_URL__` placeholder that corresponds to the configured url that gives a few benefits
- much faster and less memory intensive output transformations through not needing to parse html or markdown - the transform can be achieved using a straightforward regex find+replace
- ability to change to/from or rename subdirectory without any manual updates to the database
- modified existing 4.0 url-transformation migration rather than adding another one and repeating the transformation on posts rows
refs https://github.com/TryGhost/Team/issues/525
We expect the member event tables to be large, and they contain what is
considered metadata. For this reason we do not want to include them in
the export of sites.
refs: https://github.com/TryGhost/Team/issues/510
- In the case that host config is provided, keep staff users within the limiti
- The definition of a staff user is a user with a role other than Contributor, and whose status is not inactive
- Contributors don't count
- Suspended (status inactive) users don't count
- Locked users DO count
- Invited users DO count
- You can't invite more staff users whilst there are pending invites
- You can't unsuspend a user, or change the role on a user in such a way as will take you over your limit
- You can't import staff users - all imported users are automatically set to Contributors
- As part of this work, we are changing the default Ghost user to a Contributor otherwise it uses up a staff user
Note: there is one known active bug with this commit.
- Assume you have one remaining user within your limit. You send an invite, this works.
- You cannot "resend" that invite, it will think you're sending a new invite and hit the limit
- You must "revoke" that invite first, and create a new one
- This bug exists because the resend function uses the add endpoint & does a delete+add, but this hits the permission check before the delete
refs https://github.com/TryGhost/Team/issues/513
refs https://github.com/TryGhost/Team/issues/477
- We have skipped work to improve the 'defaultTo' value when working on Ghost 4.0 release, so adding this comment while context loaded
- defaultTo should not be set to anything as it leads to more maintenance work during major version bump
- having validation might make sense but could lead to similar maintenance work unless it's linked to some global nosion of "supported API versions" used everywhere
no-issue
The slack setting in 1.x did not have a `username` property. When
updating from 1.x, the migration to move the slack setting from a JSON
blob to two individual settings, would assume that the setting in the
database would have a value for the `username` property. This resulted
in errors on SQLite "sqlite does not support inserting default values."
The fix here is to add defaults when reading from the database, meaning
that we will _never_ attempt to insert `undefined`
refs https://github.com/TryGhost/Team/issues/476
* Moved paid subscription events population migration
This migration relies on the members_stripe_customers and
members_stripe_customer_subscriptions tables having no orphaned records
in order for it to correcly generate its data.
The migration to clean up orphaned records in those tables has not been
implemented yet, moving this migration free's up the "14" slot
* Removed orphaned stripe data from SQLite3
SQLite databases do not handle removing orphaned stripe records after a
member has been deleted. Our migration to populate the paid subscription
events relies on each customer and subscription being associated with a
member.
refs https://github.com/TryGhost/Team/issues/494
The migrations in 3.1.0 which added email permissions did not add those
permissions to the roles. This means that whilst we have the permissions
in the database, only the Owner role could use any of them.
This migration ensures that the email related permissions are added to
the correct roles.
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we've introduced backwards-compatible changes to rendering in 3.0 such as srcset and sizes which will only have taken effect on posts created or edited since the changes were made
- 4.0 brings additional changes such as image card width/height
- re-generating the `html` field of all posts from the `mobiledoc` brings all content up to latest rendering output
refs https://github.com/TryGhost/Team/issues/221#issuecomment-759105424
- Mailgun responds to an email send with a provider id in the format `<x@y.com>` but everywhere else it's used in their API it uses the format `x@y.com`
- updates email batch save to strip the brackets, and migration removes brackets from existing records so we no longer have to add special handling for the stored id any time we use it
refs https://github.com/TryGhost/Ghost/issues/12496
- having the logging require here means that workers wanting to use the db are unable to do so without requiring logging as a side-effect
- `connection.loggingHook` does not appear to be widely used for anything outside of specific debugging scenarios when using MySQL so it should be safe to disable until a proper fix is found for workers+logging leaking file descriptors
refs https://github.com/TryGhost/Ghost/pull/12598
- This changeset adds idepmotence to situations where unique contraint has to be dropped or added to the table
- Note '4.0/07-alter-unique-constraint-for-posts-slug.js` was migration that was effected by lack of idempotence
refs https://github.com/TryGhost/Ghost/issues/10318
- Storing JSON object in settings has caused multiple bugs in the past and was considered an antipattern
- This is a last bit in the long process of getting rid of "object" settings in the database. At this point there should be no settings with this type. Yey!
- This commit removes the old boot process and any files that are no longer needed as a result
- Remove the duplicate event for triggering inactive themes to load
- Tidied up a few other bits
no issue
- all migrations should contain a logging message
- info for successful things, or warn if we deviated from what was
expected for some reason
- also added some spacing to make them easier to read
refs https://github.com/TryGhost/Ghost/issues/10318
- API changes introduced:
canary/v4 Admin API
GET /settings/ (browse)
+ "unsplash" present in response as boolean value
GET /settings/:settingName (read)
+ "unsplash" present in response as boolean value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts ONLY boolean format
v3 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts either boolean or object formats
v2 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts object format
refs https://github.com/TryGhost/Ghost/pull/12631
- Initialy writen migration would break if executed twice
- Also accounted for situation if only part of the migration executed leaving only one of the values migrated
refs https://github.com/TryGhost/Team/issues/332
- The last value that has been used in the code was "members"
- By default members will be always "on" starting Ghost 4.0, so there's no need for this flag anymore
- Therefore there's no real need to keep "labs" around
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we added robust url transformation in Ghost 3.0 which ensures newly created or edited content is stored with relative URLs in the database
- this migration loops over all posts (and posts_meta) in the database performing the absolute->relative transformation to normalise data for old posts
refs https://github.com/TryGhost/Ghost/issues/12646
- adds `ghostVersion` property to mobiledoc of all posts so that we can pin the rendering output of existing content to protect against future breaking changes
- cleans any deprecated markdown card names from old Ghost 1.0 content
- cleans up HR cards that had an [accidental payload added](610d801bcf)
refs https://github.com/TryGhost/Ghost/issues/12646
- if the version is missing then the content was created before Ghost 4.0
- setting the version to `'3.0'` means it will continue to use the same rendering output so there are no unexpected breaking changes when migrating content
refs https://github.com/TryGhost/Ghost/issues/12602
* Updated members_status_events table
By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range
* Updated MEGA to handle addition of 'comped' status
With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"
* Updated customQuery for MemberStatusEvent
With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.
* Populated members_status_events table
As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.
The assumptions used in backfilling is that a Member's current status,
is their only status.
refs https://github.com/TryGhost/Ghost/issues/12602
As part of collecting Member event data, we have added a third status
for members "comped" - this fixes the population of the column to handle
this
refs https://github.com/TryGhost/Ghost/issues/12646
- `card-markdown` support is being removed from the default mobiledoc cards so we need to make sure imported content doesn't fail
- removed long-unused `cardName` payload property from tests
refs https://github.com/TryGhost/Ghost/issues/10318
- Object format used in previous "slack" setting was considered an
anti-pattern. Flag structure of separate slack_url and slack_username
values was extracted out of the "slack" JSON.
refs https://github.com/TryGhost/Ghost/issues/12602
As we only want to add events for when an email is changed, we need to store the previous email if we want a complete log of all emails for a member
refs https://github.com/TryGhost/Ghost/issues/12602
* Added members_payment_events table
This table will store successful and unsuccessful payment attempts, and
can be used to calculate gross volume over time.
* Added members_login_events table
This table can be used to audit member logins
* Added members_email_change_events table
This table will allow us to store a history of email addresses associated with a member
* Added members_status_events table
This table will allow us to track the change in status over time for members, as well
as calculate aggregates over time, e.g. paid members over time
* Added members_paid_subscription_events
This table will allow us to track subscriptions changes for members, as well as
calculating MRR over time
refs https://github.com/TryGhost/Ghost/issues/12567
- Changing unique constraint from slug to slug+type should allow for posts and pages to be created with the same slug
- The constraint will be present on application layer for API v4 while we figure out how to deal with it in API v5
Background:
- Ghosts existing boot process is split across multiple files, has affordances for outdated ways of running Ghost and is generally non-linear making it nigh-impossible to follow
- The web of dependencies that are loaded on boot are also impossible to unpick, which makes it really hard to decouple Ghost
- With 4.0 we want to introduce a new, linear, simpler, clearer way to boot up Ghost to unlock decoupling Ghost into much smaller pieces
This commit:
- adds a new ghost.js file which switches between boot mode with `node index` or `node index old` so that if we find bugs we can work around them this week
- Note: the old boot process will go away very soon, but ghost.js will remain as the interface between the command to start Ghost and the application code
- reworks the database migration process into a standalone utility, so that the DB is handled as one simple step of the boot process, decoupled from everything else
- is missing tests for this new db utility
- leaves a lot of work to do around loading core code, services, express apps in a sensible order, as work to fix this would start to break the old boot process
- doesn't use the new maintenance app because we aren't restarting the server here, instead we have the concept of a "core app" that starts in maintenance mode - need to think about how apps will be decoupled in the near future
refs https://github.com/TryGhost/Ghost/issues/12602
This allows us to backfill the data for testing purposes as well as for
use by existing sites. The mechanism for determining the date of an
unsubscribe event is crude, we use the date of the last update, which we
might expect to be _sometimes_ correct, but not always. We store the
`source` as NULL for unsubscribes as we cannot determine that with any
certainty whatsoever.
refs https://github.com/TryGhost/Ghost/issues/12567
- The method was using unneeded lodash dependency and was too complex for what it was doing
- Reshuffled internal code to use native JS filter/forEach iterators
refs https://github.com/TryGhost/Ghost/issues/12567
- Introduced here @@UNIQUE_CONSTRAINTS@@ notation allows to create unique contraints over multiple database fields. This will be needed to change posts' table unique constraint from `slug` to `slug+type`.
- The notation is equivalent to SQL's: UNIQUE(column_name1,column_name2)
- Example use in schema:
posts: {
slug: {type: 'string', maxlength: 191, nullable: false},
type: {type: 'string', maxlength: 50, nullable: false, defaultTo: 'post', validations: {isIn: [['post', 'page']]}},
'@@UNIQUES@@': [
['slug', 'type']
]
}
refs https://github.com/TryGhost/Ghost/issues/12565
- Dropping tables happens sporadicaly, usually during major version releases. It made sense to create a utility based on previous migrations of this type (e.g.: 3.0/03-drop-client-auth-tables.js migration) to avoid code duplication in the future
no-issue
This value is nullable from the Stripe API so we're making sure that we
can store exact values locally.
SQLite3 does not supports altering columns so instead we have to:
1. Create a temp table and copy the data to it
2. Delete the original table
3. Recreate the original table with the necessary modifications
4. Copy the data from the temp table
5. Drop the temp table
refs #12160
This flag will allow us easier filtering of members via the API
* Added status column to members table
This flag will be used to determine if a member is free or paid, rather
than relying on joins with the customers and subscriptions tables.
* Added migration to populate members.status
As we add the column with a default value of "free" we only need to care
about the paid members here. We also preemptively handle migrations for
SQLite where there are > 998 paid members.
closes https://github.com/TryGhost/Ghost/issues/12449
We’re starting to bump into errors with our current exporter due to the size of some of the tables in the db and hitting an issue with Ghost running out of memory during export. The intention for the export/import is not to be backup/restore functionality, but for exporting content and authors.
In addition, exporting and re-importing Stripe secret/publishable keys can cause unexpected side-effects for sites that can has major side-effects. This change -
- Removes `email_batches` and `email_recipients` tables from export data to reduce export size due to large amount of analytics data
- Removes stripe's secret/publishable/webhook keys to avoid unexpected issues with re-import
no refs
Adds new FirstPromoter settings similar to amp, which allows sites to take advantage of FirstPromoter to launch their own member referral program natively.
- Adds new firstpromoter settings group
- Adds `firstpromoter` setting to group
- Adds `firstpromoter_id` setting to group for FirstPromoter referral tracking id
- Updated tests
closes#12181
refs https://github.com/TryGhost/Ghost/pull/12265
- The 'core' value is invalid for settings 'type' column
- The 'core' default value for 'type' column in model is also invalid
- Both need to be removed as they are never used and only introduce confusion into the codebase
refs https://github.com/TryGhost/Ghost/issues/12461
- adds `members.email_count` and `members.email_opened_count` columns to contain cached counts for faster queries when outputting member data via API
- adds migration to populate cached counts with existing data
- tested locally on ~50k members which took ~4sec on mysql
- updates members output serializer to include the new fields in API output
refs https://github.com/TryGhost/Ghost/issues/12421
- nullable so we can distinguish between members that have and haven't received any trackable emails
- indexed because we'll be using this column for sorting
no issue
- cleans up unused tables `emails.{meta,stats}`
- adds timestamp columns `email_recipients.{delivered_at,opened_at,failed_at}` that can be used for event timelines and basic stats aggregation
- indexed because we want to sort by these columns to find the "latest event" when limiting Mailgun events API requests
- adds aggregated stats columns `emails.{delivered_count,opened_count,failed_count}`
- adds a composite index on `email_recipients.[email_id,member_email]` to dramatically speed up `email_recipient` update queries when processing events
- modifies the db initialisation to support an `'@@INDEXES@@'` key in table schema definition for composite indexes
no refs
- Migrations were using previous default for badge which was set to `false`
- Default config for badge was `false` which caused migration to switch off the badge, updated to `true`
- these helpers remove a lot of the duplicated code that we had when doing up/down column migrations, and provides a much shorter way of doing this in the future
no issue
Updated newsletter badge to be shown by default which was previously set to be hidden. Users can still control the visibility via Email settings in Ghost-Admin
no-issue
We want to give users to ability to customise the content of their newsletter, and the first step
toward that is a setting in which we can store text or html to embed in the template
no-issue
* Handled send_email_when_published in Posts API
This restores backwards compatibility of the Posts API allowing existing
clients to continue to use the `send_email_when_published` flag. This
change uses two edits, which is unfortunate. The reason being is that
this is an API compatibility issue, not a model issue, so we shouldn't
introduce code to the model layer to handle it. The visibility property
of the model is used to determine how to fall back, and because it can
be left out of the API request, and relies on a default in the settings,
we require that the model decide on the `visibility` before we run our
fallback logic (or we duplicate the `visibility` default at the cost of
maintenance in the future)
* Dropped send_email_when_published column from posts
Since this column is not used any more, we can drop it from the table.
We include an extra migration to repopulate the column in the event of
a rollback
* Updated importer to handle send_email_when_published
Because we currently export this value from Ghost, we should correctly
import it. This follows the same logic as the migrations for this value.
* Included send_email_when_published in API response
As our v3 API documentation includes `send_email_when_published` we must
retain backward compatibility by calculating the property.
* Fixed fields filter with send_email_when_published
* Added safety checks to frame properties
Some parts of the code pass a manually created "frame" which is missing
lots of properties, so we check for the existence of all of them before
using them.
* Fixed 3.1 migration to include columnDefinition
We require that migrations have all the information they need contained
within them as they run in an unknown state of the codebase, which could
be from the commit they are introduced, to any future commit. In this
case the column definition is removed from the schema in 3.38 and the
migration would fail when run in this version or later.
refs https://github.com/TryGhost/Ghost/issues/12355
- Adds new default settings for newsletter customisations - `newsletter_show_badge`, `newsletter_show_header` and `newsletter_body_font_category`
- Adds migrations to update group for new settings
- Add migration to update settings based on existing config value for newsletter settings
- Passes new newsletter settings to newsletter template and updates design based on them
- Fix tests
- this check should never really be hit unless the DB has missing data,
but the code tries to read from an undefined variable and crashes, which we
can protect against
- this can be triggered by doing a `knex-migrator init` followed by a
`knex-migrator rollback --v 3.0 --force`
- 504509bb6 removed the global override for Promise
- there are a bunch of places in code that use Bluebird Promise methods,
but Bluebird wasn't being imported in these places
- this would have thrown errors all over the place
no issue
- tracking of bulk email opens can be enabled/disabled over time, if we're calculating analytics for emails we don't want emails which didn't have tracking enabled skewing the results so we need a record of whether tracking was enabled for each email
no-issue
This column will allow us to store the canonical recipient filter on the
email resource giving us a detailed log of which members an email was
intended for
no-issue
This column will allow us to decouple the recipients of newsletter from
the `visibility` of a post, allowing us to send emails to specifically
free members, or to send paid posts as newsletters to all members.
no refs
[Portal](https://github.com/TryGhost/Portal) is a new drop-in script to make the bulk of Ghost membership features work on any theme out of the box, which was under a developer flag so far. This release removes the flag for Portal and makes it included as default for any members-enabled Ghost site. The Portal script is backward compatible with old public members script and existing Members-enabled themes should notice no change.
- Removes Portal config flag as Portal is now enabled by default
- Removes old members script as Portal is backward compatible with it
- Changes `{{content}}` helper to show default CTA in case of restricted content access
- `accent_color` setting is no more behind the dev experiment flag and included by default
- Adds migration to switch off Portal button setting for all existing sites which don't have Portal enabled in beta
no issue
- all staff users can browse/read snippets so they're usable in the editor for everyone
- only administrators, editors, and admin integrations are able to create/edit/delete snippets
no issue
- minimal table structure required for the first iteration of content snippets
- snippets are stored pieces of re-usable content that could effectively be entire posts so the `mobiledoc` field length matches the `posts.mobiledoc` field length
no issue
- Member email settings - newsletter and support - was previously being set to null for all cases, which can lead to `null` values being set for those addresses on a successful import whereas the expected behavior is to retain the old values.
- Adds the 2 email settings to ignored settings list so they are not changed during import process
no issue
The email table should be a reference for all data that was used when sending an email. From and Reply-to addresses can change over time and we don't have any other reference for their value at the time of sending an email so we should store them alongside the email content.
- schema updated with `from` and `reply_to` columns
- both are set to `nullable` because we don't have historic data (can be populated and changed in later migrations if needed)
- neither `from` or `reply_to` have `isEmail` validations because they can have name+email in an email-specific format
- will help keep concerns separated in the future. `mega` service can deal with all of the email contents/properties, and the `bulk-email` service's concerns are then only email sending and any provider-specific needs
no-issue
After discussion with Matt, we decided that 192 bits for the token is a
good number, as it has no padding when base64 encoded and is more secure
than 128 bits, whilst still a managable size.
no-issue
This is a table to store single use tokens for use in magic links, the
columns are as simple as possible at the moment and are designed as:
id - standard ObjectID like all of our tables
token - 128bit base64 encoded string
data - arbitrary data to store against the token
created_at - timestamp to allow for expiry to be implemented for tokens
no issue
We want to store a list of recipients for each bulk email so that we have a consistent set of data that background processing/sending jobs can work from without worrying about moving large data sets around or member data changing mid-send.
- `email_batches` table acts as a join table with status for email<->email_recipient
- stores a provider-specific ID that we get back when submitting a batch for sending to the bulk email provider
- `status` allows for batch-specific status updates and picking up where we left off when submitting batches if needed
- explicitly tying a list of email recipients to a batch allows for partial retries
- `email_recipients` table acts as a join table for email<->member
- `member_id` does not have a foreign key constraint because members can be deleted but does have an index so that we can efficiently query which emails a member has received
- stores static copies of the member info present at the time of sending an email for consistency in background jobs and auditing/historical data
refs #11999
- The `routes_hash` setting will be used during the boot process to update the hash
of currently loaded routes.yaml file in case it's different from last restart
no issue
- Updated magic link generation and validation methods for email update API to handle new support address
- Updated importer to ignore the new support address as it can only be updated via verification
- Updated members service to listen on settings edit for new support/reply address fields as well
- Updated tests to include the new settings
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account, public setting
- `members_reply_address` - Where you receive responses to newsletters
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account
- `members_reply_address` - Where you receive responses to newsletters
refs #12126
- Adds migration to add impersonation permission to administrators
- Adds default permission fixture to allow administrators to read member impersonation urls
- Allows administrators to create member impersonation magic links
closes https://github.com/TryGhost/Ghost/issues/12129
Migrations were failing when upgrading from a version that didn't include stripe member subscriptions table. The `up` migration was checking for the existence of a unique index that was added to the schema in 3.29 but it was using the wrong variable name which meant that it would never return true for an existing index. For most migrations this worked because the index existed but when through 2.34 the table was created from scratch and did included the index at that point.
- fixed variable name and re-ordered variable assignment for better code locality that would have made the typo more visible
- deleted files under `core/server/lib/promise` and related test files
- added `@tryghost/promise` as a dependency
- fixed all local requires to point to the new package
no issue
For large numbers of members we're unable to perform queries for free/paid members in a reasonable time without indexes. Bulk delete is also very slow when looping through bookshelf models and having bookshelf manage deletion of child records, adding `ON DELETE CASCADE` to the foreign key indexes moves child deletion to the DB level and allows for performant bulk delete queries.
- migrations only run on mysql because sqlite does not support altering tables
- adds foreign key indexes and constraints with 'ON DELETE CASCADE' for members_labels, members_stripe_customers, and members_stripe_customers_subscriptions
refs #12100
For performance reasons we want to add foreign key and unique constraints
to the members_stripe_* tables so we can utilised cascading deletes and
joins across the tables when querying.
In order to do this we must first ensure that:
- There are no duplicate entries in the `subscription_id` or `customer_id` columns
- There are no orphaned rows in the subscription or customers tables
If the first is not true, the unique constraint will fail, and if the second is not true,
the foreign key constraint will fail.
As we are only adding the indexes to existing MySQL databases at this point, the
cleanup migrations will also only be done for existing MySQL databases too.
The migrations for removing orphaned rows splits the deletion into a `SELECT`
followed by a `WHERE IN` to avoid the database "optimising" the query into a
`JOIN` which ends up taking much longer due to the lack of indexes.
no-issue
We are in the process of creating migrations to add foreign key constraints
and cascading deletes to the members_stripe_* tables to make listing members
and deleting members faster. As well as the migrations we need to update the
database schema so that new installations have the correct indexes and constraints.
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
closes#12001
* Moved settings validation to the model
This moves the settings validation out of the validation file and into
the model, as it is _only_ used there.
It also sets us up in the future for custom validators on individual
settings.
* Improved validation of stripe_plans setting
- Checks `interval` is a valid string
- Checks `name` & `currency` are strings
* Moved stripe key validation into model
The stripe key settings are all nullable and the regex validation fails
when the input is `null`. Rather than reworking the entirety of how we
validate with default-settings validation objects, this moves the
validation into methods on the Settings model.
* Added tests for new setting validations
Adds tests for both valid and invalid settings, as well as helpers
making future tests easier and less repetitive
no-issue
They will be used to store webhook information so that we can persist it between
boots and simplify the creation process of webhooks in members
closes https://github.com/TryGhost/Ghost/issues/12026
- 3 new portal settings were added in `3.23` in default settings - `portal_button_style`, `portal_button_icon`, `portal_button_signup_text`
- New settings default to group `core` on migrating from pre 3.22 versions due to missing columns in DB
- Migration here updates the new settings to correct group
no issue
- Existing and new sites should have portal button hidden by default and need explicit switch on from Admin.
- Default value for new sites has been updated, but existing sites still have old default and portal button set as true
- Migration here resets portal button setting to false for all existing sites, and they'll need to be explicitly turn it on from admin again
no issue
- Adds new portal settings - `portal_button_style`, `portal_button_icon` and `portal_button_signup_text`
- New settings allows customization of portal button
- Updates tests to include new settings
* tag '3.22.2':
v3.22.2
Updated Ghost-Admin to v3.22.2
Emitted all settings events on reinit of cache (#12012)
🐛 Updated access to be true by default in v3 API
Hardened members subscription migration against missing data (#12009)
closes#11993
We had some issues with some databases being in an unexpected state this
check for each property before using it add uses defaults when it is
missing.
no issue
- The membersjs default trigger button was set to be visible by default, even behind dev flag
- This changes the default visibility to false so it needs to be switched on from Admin to be visible on screen
refs https://github.com/TryGhost/Ghost/issues/10318
- Skipped `members_subscription_settings` and `stripe_connect_integration` from settings import
- Fixed import data field filtering for core and theme groups
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates default settings to contain correct type and validation for each setting
- Updates `populateDefaults` to correctly insert type value for new settings
- Updates settings schema to allow only select types - `array`, `number`, `boolean`, `string`
- `object` is a temporary type allowed till we get rid of all JSON object settings
refs #10318
Because settings are not populated with the correct group and flags, we
must _always_ set these. Then we can check to see if there are values
which need migrating, and if not, can safely exit and leave the values
as default.
refs #10318
If migrating from a previous version that does not include the setting
being migrated from we can safely not update the new setting, and just
rely on its default value being present. When rolling back we can use
defaults if the new setting does not exist.
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates the type field for a setting to specify its type explicitly
- Type can be one of (string, number, boolean, array)
- `object` is allowed a temp type till all the object settings are removed
- existing `type` value for setting was misleading and referring to `group`
- we moved the current `type` value for a setting to `group` here - 4a9e57c170
* Updated members default settings
ref #10318
This pulls out the members_subscription_settings & stripe_connect_intgration settings into separate keys
* Updated usage of members_from_address
* Updated stripe_connect usage
* Updated members config to use new settings
* Updated members middleware to use isStripeConnected
* Updated members service to reload correctly
We reload the members-api instance when the related settings change, so
this makes sure we're listening to the correct settings changes
* Updated ghost_head helper to use new settings
* Updated theme middleware to use new settings
* Renamed members_allow_signup -> members_allow_free_signup
* Fixed tests after settings refactor
* Removed from direct key settings key
* Fixed regression tests for settings api
refs #10318
As populateDefaults is run _before_ migrations, the new settings will
already be inserted in the database, so we just need to update their
values and then delete the old settings.
refs https://github.com/TryGhost/Ghost/issues/10318
`Settings.populateDefaults()` is run before migrations during Ghost's startup. This can cause problems when new settings table columns are added (and populated in `default-settings.json`) because `populateDefaults()` was using the model layer which assumes that those columns are available, resulting in `ER_BAD_FIELD_ERROR: Unknown column` type errors.
- query the database for the available `settings` table columns
- switch to using raw knex queries without Bookshelf for insertions so that we're in control of the columns that are added
- use `_.pick` to skip any properties in `default-settings.json` that do not match to an available column - those columns will be added and populated by later migrations
- moving away from using the model to insert settings has the side-effect of not emitting `settings.added/edited` and `settings.x.added/edited` events, this should be fine because `populateDefaults()` is called before anything else is set up and listening
- added a call to `populateDefaults()` in our knex-migrator "before migration" hook so that we have consistent db state across both startup initialised migrations and manually triggered knex migrations
no-issue
These utils are the first steps toward getting the models out of our
migrations! The utils here interact directly with the database and where
possible don't reinvent the wheel, by using smaller building blocks to
build more comples ones.
refs https://github.com/TryGhost/Ghost/issues/10318
- `shared_views` is always an array (Ghost-Admin has handling to revert it to an array if it's anything else) but it's default value was set to an empty object
refs https://github.com/TryGhost/Ghost/issues/10318
- maps old `settings.type` values to new `settings.type/group` values
- uses an explicit map so that we don't lose information and can safely roll back even though we're modifying `settings.type` too
- updates `settings.type` values too to keep code working while we switch to using `settings.group`
- sets the `settings.group` value for all settings which are keeping the same group as their current type
- adapts `settings.type` validations to match new groups
- adds flags to specific settings, both in the migration for existing settings records and in default-settings.json for new settings records
refs https://github.com/TryGhost/Ghost/issues/10318
- `group`
- to replace the `type` column, provides a more descriptive name for the columns use
- for existing sites it will be populated by migrating data from the `type` column in a later migration
- for new sites a minimal update has been added to `parseDefaultSettings()` to populate the `group` field when settings are created during startup - fixes the NOT NULL constraint on `settings.group`
- `flags`
- signifies special handling that is different to other settings in a group
- eg, `PUBLIC,RO` would indicate that the setting is available via unauthenticated endpoints and is read-only
refs #10318
refs 2614565d5a
- Adds importer mapping for fields as in referenced migration
- The intention is to allow exports from v2/v3 to still be compatilbe with current version
- Adds a mapper for deprecated fields and imports them with new settings "keys"
refs #10318
refs 2614565d5a
- Renamed ghost_head/ghost_foot in settings to match the new names
introduced in migrations
- Above change lead to reshufling in the mappings in input/output
serializers
- Makes sure change is compatible with v2 API
refs https://github.com/TryGhost/Ghost/issues/10318
refs 2614565d5a
- Renames to match referenced migration renames
- Fixed API responses so they are consistent with newly renamed fields
- Not returning lang and timezone keys from settings in API v2 ther rest should be returned in API v3/canary
