- renamed our internal validation library to "validator" - which is the same as the tool it wraps
- updated the public api so that validator methods are directly exposed
- this will make it a drop-in replacement for validator-js
- in turn, this allows us to pull this out into @tryghost/validator, and use our own wrapper instead of the 3rd party library
- General code cleanup
- Removed unused notContains rule
- Swapped custom empty rule for builtin isEmpty rule
- Dropped usage of .extend on validator, as this was removed 2 years ago!
- This will allow us to upgrade the validator dependency to a much newer version
- Changed our internal validator module to only expose the functions we use.
- This gives us a clearer Public API
- It makes it easier to see if we are affected by changes in validator
- It's still easy to add another validator, we just have to update what we require
- We can potentially use this to make smaller builds esp for client-side usage
- Once ripped out into a module we can use ES imports :D
- Rejigged and _slightly_ improved the tests
- This is super specific code relating only to validating passwords.
- It's needed as a shared validator as we use other funnels to help people setup Ghost on Pro, but currently it's hard-baked into Ghost
- It's also not the greatest code. It'd be nice to be able to rework it and know that would automatically update everywhere passwords are set
- This is a really specific piece of code related to validating models against our internal schema.js format
- This doesn't make sense without a schema.js file
- It does depend on the internal validator and validate tools - but those are used elsewhere too, and can reasonably be moved out of the codebase
- I don't see schema.js moving out of the codebase any time soon. We can move the validator but it would be a class that requires schema via DI
- For now my focus is on getting the data/validation tooling separated and making clear sense
- Improving data/schema can come later :)
- The data/validation module is made up of several loosely related things with lots of dependencies
- Separating out the various components makes it possible to see what's what, and importantly what has complex dependencies
- validator + validate probably go togetheri in an external module, the other two files should probably have their own homes in related areas of ghost e.g. schema -> data/schema/validate.js
refs https://github.com/TryGhost/Team/issues/770
We want post feature image functionality to better match what's available inside the editor, to do that we'll need somewhere to store alt and caption meta data. `posts_meta` chosen because even though we want to make this generic for other tables in the future those tables also have a `feature_image` (or closely related) field.
- updated schema with new columns
- added migration to create columns
- cleaned new columns from API output
- not output on v2/v3
- conditionally output on v4/canary output based on labs flag
- bumped `@tryghost/admin-api-schema` to allow new columns through in canary API requests
- silently clean properties from input when labs flag is disabled
- updated acceptance tests so they fail if `admin-api-schema` is not letting the new fields through
no issue
Shows impact of new code behind labs flags through the existing acceptance/regression tests. Allows for existing tests to be updated to match new behaviour rather than requiring separate tests where individual flags are enabled. Should result in minimal test updating once code reaches GA.
- adds a forced `'labs:enabled'` fixture op that edits the `labs` setting to enable all flags then restarts the settings service to pick up the new setting
- modifies labs service to not remove ALPHA_FEATURE labs settings when running in a testing environment
refs https://github.com/TryGhost/Team/issues/757
refs https://github.com/TryGhost/Team/issues/332
refs ea6d656457
- We have a need a quick way to add features behind flags. The old way of "labs" is the quickest way to achieve this. It has ready tooling around it and well understood pitfalls. This change reintroduces "labs" group & key in settings table in the same shape it used to be (see reffed commit)
- Next step will be introducing very basic guard rails to protect from pitfalls previous implementation of "labs" had. This will include an allowlist based input validation for lab's object's data
- The labs being an "object" type is an EXCEPTION. Even though it's an antipattern we aim to move away from, for now it's the lowest impact solution that will unblock the use of flags in the system. A proper solution will come at some point.
refs https://github.com/TryGhost/Team/issues/710
refs https://github.com/TryGhost/Team/issues/725
Products will now have a single monthly and yearly price which will be
used throughout Themes, Portal & Admin. These columns will be used to
track the current prices for each of them, and will update anytime we
change the pricing of a product.
Due to a circular table dependency we have not added a foreign key
constraint to the new columns, this will be handled at a later date. It
is tracked in issue 725 references above
no issue
Our server-defined `mobiledoc` object was required by `UrlUtils.cardTransformers` property to help set up a shortcut for url transform functions but that was breaking the independence of `UrlUtils` by crossing the shared/server boundary.
- `cardTransformers` is only needed for the `mobiledocToTransformReady` utility function that will only be used by the server
- removed `UrlUtils.cardTransformers` (and associated require) from our `UrlUtils` instance and updated the few areas the server uses `mobiledocToTransformReady()` to pass in the mobiledoc card objects directly as an option
refs https://github.com/TryGhost/Team/issues/635
This is to ensure we don't break migrations for any sites which have
imported external subscriptions which have an interval of 'week' or
'day'
The bump to members-api includes the handling of these intervals for
ongoing population of mrr events
refs https://github.com/TryGhost/Team/issues/610
- Before introducing a new test for the refed issue doing a linting cleanup. The result will be removing one of `File has too many lines ` lint warnings
refs https://github.com/TryGhost/Team/issues/693
Since we've got rid of the concept of Complimentary with the Custom
Prices work, we're removing the 'comped' status from members. This
involves a migration for existing members, a schema update for the
validation, and a bump to members-api to no longer use the 'comped'
status for new members.
We also update the aggregation of the MemberStatusEvent to consider the
'comped' status as 'paid', and that there are 0 'comped' status events
in the database.
We can consider a migration for this data in the future, either adding
new status events moving from 'comped' to 'paid', or by modifying
existing status events. However both of these are very difficulty to
write a down migration for, and might be best saved for a major version.
- @tryghost/members-api@1.7.0 is the version that includes the required
changes, however we have already bumped to 1.8.0 in Ghost
no refs
Since backend now allows multiple prices but we want the prices to be currently limited to monthly/yearly on UI, we need new settings to store the current monthly/yearly price by the site owner. These settings determine the active prices shown in Admin / Portal for the site till we allow all custom products/prices again.
refs https://github.com/TryGhost/Team/issues/581
closes https://github.com/TryGhost/Team/issues/582
Emails can now be sent to members with specific associated labels or products by specifying an NQL string. We want to bring the same members segment feature to content by allowing `visibility` to be an NQL filter string on top of the `public/members/paid` special-case strings.
As an example it's possible to set `posts.visibility` to `label:vip` to make a post available only to those members with the `vip` label.
- removed enum validations for `visibility` so it now accepts any string or `null`
- bumped `@tryghost/admin-api-schema` for API-level validation changes
- added nql validation to API input validators by running the visibility query against the members model
- added transform of NQL to special-case visibility values when saving post model
- ensures there's a single way of representing "members" and "paid" where NQL gives multiple ways of representing the same segment
- useful for keeping theme-level checks such as `{{#has visibility="paid"}}` working as expected
- updated content-gating to parse nql from post's visibility and use it to query the currently logged in member to see if there's a match
- bumped @tryghost/members-api to include label and product data when loading member
refs https://github.com/TryGhost/Team/issues/667
On clean and existing installs, the default product created should be named the same as the site title in the first setup so the UX on Portal and everywhere is consistent. This change adds a migration to update existing sites which already have a default product created via fixture, and rename them to their current site title. The rename is only done if the Product name is still the same as in fixture - `Default Product`.
refs https://github.com/TryGhost/Team/issues/671
When turning on custom products, existing sites should have default price descriptions that match existing values for prices. This change sets the default description for Free price to match existing hardcoded value.
no-issue
Our base model will only automatically convert numbers to booleans if
the type is 'bool' - however this column was incorrectly added with a
type of 'boolean'. Lucklily - knex with both MySQL & SQLite3 will add
a column with the same type for both of these, so no migration is needed
to fix it.
refs https://github.com/TryGhost/Team/issues/637
The "free" price - when Members signup without using Stripe, should have
a name and description, so that it can be displayed in Portal in a
similar way to paid price's. As there is only ever one, and it is not a
fully fledged price, a setting makes more sense than a dedicated db
table.
refs https://github.com/TryGhost/Team/issues/586
We are no longer using the `stripe_plans` setting, instead we are using
the `stripe_prices` database table. However, we must keep the setting as
the migration from the setting to the database is not done as a standard
migration, but in code. This means our code has to still read and pass
the setting because we will never know if the migration in code has run
yet.
The `portal_plans` setting has been updated to only include 'free' by
default, because the setting must include id's now rather than names.
refs https://github.com/TryGhost/Team/issues/586
A discussion in the Members team resulted in us determining that we do
not need to enforce unique names for Products. Stripe does not enforce
uniqueness for their Products, and we feel it's not necessary for us to.
refs https://github.com/TryGhost/Team/issues/581
refs https://github.com/TryGhost/Team/issues/582
When publishing a post via the API it was possible to send it using `?email_recipient_filter=all/free/paid` which allowed you to send to members only based on their payment status which is quite limiting for some sites.
This PR updates the `?email_recipient_filter` query param to support Ghost's `?filter` param syntax which enables more specific recipient lists, eg:
`?email_recipient_filter=status:free` = free members only
`?email_recipient_filter=status:paid` = paid members only
`?email_recipient_filter=label:vip` = members that have the `vip` label attached
`?email_recipient_filter=status:paid,label:vip` = paid members and members that have the `vip` label attached
The older `free/paid` values are still supported by the API for backwards compatibility.
- updates `Post` and `Email` models to transform legacy `free` and `paid` values to their NQL equivalents on read/write
- lets us not worry about supporting legacy values elsewhere in the code
- cleanup migration to transform all rows slated for 5.0
- removes schema and API `isIn` validations for recipient filters so allow free-form filters
- updates posts API input serializers to transform `free` and `paid` values in the `?email_recipient_filter` param to their NQL equivalents for backwards compatibility
- updates Post API controllers `edit` methods to run a query using the supplied filter to verify that it's valid
- updates `mega` service to use the filter directly when selecting recipients
refs https://github.com/TryGhost/Team/issues/581
Editors are allowed to restrict post visibility and send emails to particular member segments, they need to be able to read labels so that they can select them in a member segment.
refs https://github.com/TryGhost/Team/issues/496
We want to give more control over the default selection of email recipients when publishing a post, to do that we need somewhere to store those settings. These settings are site-wide and intended for use by admins to control the default editor behaviour for all staff users. They _do not_ control API behaviour, if you want to send email when publishing via the API it's still necessary to explicitly opt in to that using the `?email_recipients_filter=` query param.
- new `editor` settings group to indicate that these settings only affect the UI rather than the API
- `editor_default_email_recipients` controls overall behaviour, string/enum with these allowed values:
- `'disabled'`: no option to send email is shown in the editor's publishing dropdown
- `'visibility'`: (default) selected member segment is dynamic and matches the post visibility filter
- `'filter'`: specific member filter defined in `editor_default_email_recipients_filter` setting
- `editor_default_email_recipients_filter` is an NQL string for selecting members, used when `editor_default_email_recipients` is set to `'filter'`
- default value is `'all'`
- the segment string can be any valid NQL filter with the additional special-case values of `'all'` and `'none'`
- single authors were deprecated in v1.22 when we added multiple authors
- we always thought we'd clean this up a lot sooner, but it's stuck because it's an annoying thing to break people's shit over
- still saying "remove in vX" isn't useful, we need to know how long a feature has been deprecated so we can judge whether it's safe to remove
refs https://github.com/TryGhost/Team/issues/586
The `products` and `stripe_prices` tables are missing a description
column which will be used by Portal to display information about the
products and prices
refs 829e8ed010
- i18n is used everywhere but only requires shared or external packages, therefore it's a good candidate for living in shared
- this reduces invalid requires across frontend and server, and lets us use it everywhere until we come up with a better option
refs https://github.com/TryGhost/Team/issues/579
The previous `members_allow_free_signup` -> `members_signup_access` migration made a direct correlation between the toggle `true/false` to `all/invite` under the assumption that behaviour between the two settings would be identical. The assumption was incorrect and the behaviour is changing so `invite` forces invite-only mode, stopping all front-end signup to free or paid plans with the free plan now being disabled via the portal plans setting.
- check existing `members_signup_access` setting and if it's `'invite'` migrate it to `'all'` where signup should still be possible. The "invite-only" mode should only be active if certain conditions are met:
- Stripe is not configured ("allow free member signup" off and no Stripe showed "invite-only" in portal)
- Stripe is configured but no plans are selected in portal (no plans showed "invite-only" in portal)
- when migrating `'invite'` to `'all'`, also remove `'free'` plan from the `portal_plans` setting to avoid previously paid-only sites unexpectedly showing a free plan on signup
refs https://github.com/TryGhost/Team/issues/634
- find earliest backup file created when a 4.3 migration was run, if found use the `members_allow_free_signup` value from there to change `members_signup_access` from `'all'` to `'invite'` if necessary
- Having these as destructured from the same package is hindering refactoring now
- Events should really only ever be used server-side
- i18n should be a shared module for now so it can be used everywhere until we figure out something better
- Having them seperate also allows us to lint them properly
refs https://github.com/TryGhost/Team/issues/634
- the migration moving `members_allow_free_signup` to `members_signup_access` was expecting a raw boolean setting value but the actual value is a string so always evaluated as truthy making all sites look like they had "allow free members signup" toggled on when generating the new setting's value
- updated to check for an explicit string value in `up` and set an explicit string value in `down`
refs c873899e49
- as of `bson-objectid` v2.0.0, this library exports the function
to generate an ObjectID directly, and then you need to use `.toHexString()`
to get the 24 character hex string - 6696f27d82
- this commit removes all uses of `.generate()` and replaces with this
change
refs https://github.com/TryGhost/Team/issues/619
As part of the Custom Products work, we are linking members to products
when updating their subscriptions. This requires that we have at least
one product in the database. For existing sites that are using Members
this is handled by the v4.3 03 migration. But for new sites we must
include a fixture.
Also fixes the tests to not reply on the order of the fixtures
refs https://github.com/TryGhost/Team/issues/586
- Adds new `stripe_price_id` column to subscriptions table to store stripe price ids with `index`
- Populates `stripe_price_id` column value to current `plan_id` making the `plan_*` values redundant
- Updates tests
refs https://github.com/TryGhost/Team/issues/586
If a product inside Ghost is deleted, we want to cascade delete all associated Stripe products and prices as they always need to refer back to a ghost product and will hang without any reason otherwise. This change adds cascade delete for products -> stripe_products -> stripe_prices to avoid broken states
refs https://github.com/TryGhost/Team/issues/579
Currently the members signup setting is explicitly yes/no to allowing free members signup, with the implication that when set to "no" members is still active but members have to be created via Stripe or the admin API.
This change renames the setting and changes its type to allow more than a binary option.
- migration to create/update the new setting based on the old value
- free signup = "all", no free signup = "invite"; matches the current UI for this setting
- rename setting everywhere it's used/tested against
- modify `getAllowSelfSignup()` used to configure members packages to only return `true` when the new setting is set to `'all'` to match behaviour to the older setting
- update importer to rename the setting when importing from an older Ghost version
refs https://github.com/TryGhost/Team/issues/618
- The `oauth_client_id` and `oauth_client_secret` are placeholders to store OAuths related data.
- The flag for `oauth_enabled` or anything along those lines was not added intentionally in favour of checking if the `oauth_client_id` & `oauth_client_secret` are null.
refs https://github.com/TryGhost/Team/issues/616
All roles which can publish posts should be able to read/browse products, as content gating
will be based on products going forward.
Creating, updating & destroying products will often make modifications to Stripe which requires
Administrator or Owner roles.
We also improve the permissions tests so that we no longer rely on things being in a particular
order.
refs https://github.com/TryGhost/Team/issues/586
- Change the interval column to be `nullable` as one time payments won't have any interval
- Remove the `livemode` column as we store the connected account's livemode status at top level
no issue
- There is a valid subset of statuses that can be set for the users but there's no "isIn" validation for possible values
- Additionally some of the statuses like warn-1, warn-2, etc. don't have a clear usecase (or at least nothing was found in the codebase for them to be used). They might be up for removal if this assumption is correct
refs https://github.com/TryGhost/Team/issues/586
- Add the `stripe_products` table, so that we can map Stripe Products to Products in Ghost
- Add the `stripe_prices` table, so that we can associate Stripe Prices to Products table
refs https://github.com/TryGhost/Team/issues/586
- Add the products table, so that we can store Products in Ghost
- Add the members_products table, so that we can associate Members w/ Products
- Use sort_order on the members_products table to follow the same convention in members_labels
- Populate the products table with a single product, using the name from the stripe_product_name setting
- Populate the members_products table with relations based on the status column of the members table
Populating the tables allows us to transition from the current system, which does not care about products, into the
new system, where Products are used to group members. The intention is that all existing paid members have the
same product
closes https://github.com/TryGhost/Team/issues/595
Due to a bug in `mrr_delta` calculation, we ended up reducing the MRR delta by twice the original amount when a subscription goes from active to canceled and storing it in `members_paid_subscription_events` table, which is used to show the MRR chart on Dashboard. The way we identify the incorrect events in the table which got the double negative value is by checking if they match certain criteria - Both `from_plan` and `to_plan` have same value as a subscription changes status while being on same plan.
This migration halves the `mrr_delta` for incorrect events to restore the correct MRR change for the site.
refs https://github.com/TryGhost/Team/issues/555
- Export files included a lot of data which was not used in the importer, for example: members, labels, migrations and many more. This lead to a lot of clutter in the import files and made it hard to reason about their purpose.
- The main purpose of exports - is to export importable resources. These are posts, tags, and users. The rest of data like members or migrations either have their own importer (like CSV importer for members) or does not and should not have any ways to be imported.
- These changes are in now way complete. It's a first step towards resource-based exports which could be properly versioned in the future on API level and not be a mirror of the DB structure.
- This is sort of a breaking change. But we are doing it because: (1) its an internal API that should not be used by external clients, (2) there was no public contract to have this API stable at any point, (3) we really need to get back the control over export files structure and size
- In case an external client was dependent on some structure of the exported json file they can still pass in ALL of previously exported data by passing table names in `include` query parameter.
refs https://github.com/TryGhost/Team/issues/555
- Previous blocklist approach was resulting in adding every single new table into an export automatically. Which creates possibility to leak sensitive data if not used porperly. Allowlist approach gives better control over what is exported, makes this information explicit, and version-control friendlier
refs https://github.com/TryGhost/Team/issues/555
- The getVersionAndTables was doing too much and was only used once creating clutter in doExport method. Refactored code doing 2 direct calls instead of destructuring "dances".
closes https://github.com/TryGhost/Team/issues/557
After updating to v4 on SQLite from v3, we were unable to delete members due to a foreign key mismatch error. This is because the migrations which recreates the tables for `members_stripe_customers_subscriptions` and `members_stripe_customers` doesn't add back the unique constraint which is needed by FKs for reference.
The migration creates the missing UNIQUE constraints on the tables.
- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- Fixture posts cover the key features and give users an introduction to how to use their site
- This all comes from the marketing and design teams to refresh the look and feel of Ghost and give users the best possible onboarding experience
Note: this fixture overhaul includes
- new content for new 4.0 features
- regenerated post content using our updated mobiledoc structure
- a switch from British to US English
refs 2bba9989db
- Note: this will require new fixtures so that the navigation links actually work
- These updates are all in aid of getting the best possible default setup and onboarding experinence for new Ghost users
- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- This all comes from the marketing and design teams to refresh the look and feel of Ghost
Note on accent color:
This commit changes the default accent colour again.
The intention is that new sites should get #FF1A75 (pink) as their default.
Any existing sites that do not have an accent colour set yet, should get #15171A (black) on upgrading to 4.0.
These are different as they are different experinces. Fresh sites will be guided to pick a color, so
a bright color is more visible and helps to see what can be done, whilst existing sites get a muted
black, that should be a sensible fall back color.
refs TryGhost/Team#535
We want to ensure that a site will always have a default value of
`'#15171A'` for the accent_color setting.
Since the boot process changed we have three cases to account for:
1. Setting does not exist
2. Setting exists with no value
3. Setting exists with a value
It is only in the case of 2. that we want the migration to update the
database with a default value.
In the case of 3. the site owner has already set a value, which we do
not want to override.
In the case of 1. the setting will be created (and populated with
default value) from the default-settings.json file, by the
populateDefaults method called from the settings service
We also update the accent_color setting to include a non-empty
validation, to ensure that the setting will always have a value, as
sites before 4.x may have an empty accent_color, we must update the
importer to set the default value if one is not present. Otherwise we
would run into validation errors and even if we didn't would have an
invalid database state.
no-issue
We were originally checking the state of the database, e.g. if a foreign
key constraint existed, so that we could conditionally act upon it. This
was to ensure that our migrations are idempotent.
Some database configurations, for example if you have many databases on
a single MySQL instance, would cause these information_schema queries to
take an exceptionally long time.
In order to speed up migrations, we instead attempt the action we want
to apply to the database, and then catch relevant errors to ensure the
migration is idempotent.
SQLite does not error when adding duplicate foreign or primary key
constraints, meaning that we must keep in pre-checks for these
operations, when running on SQLite
Co-authored-by: Daniel Lockyer <hi@daniellockyer.com>
no issue
Upgrading from v1 or v2 can result in successful upgrades but with mobiledoc errors showing in the logs:
```
NAME: InternalServerError
MESSAGE: Mobiledoc card 'card-markdown' not found.
```
The errors do not signify a problem as long as the 4.0 migrations run because those rename the deprecated card before re-rendering.
- `@tryghost/kg-default-cards` dropped support for `card-markdown` cards. 4.0 migrations handled this by renaming all `card-markdown` cards to `markdown` before re-generating any content
- 2.0 and 3.0 also had migrations that re-generated content but they are run before the 4.0 card rename migration meaning that the mobiledoc renderer sees cards that it doesn't know about. The behaviour for unknown cards is to log an error and skip rendering of that card
- by NOOPing the 2.0 and 3.0 migrations we eliminate the incompatibility errors and reduce the amount of processing the upgrade needs to perform
closes https://github.com/TryGhost/Team/issues/467
- switches to storing "transform-ready" URLs in the database
- transform-ready URLs contain a `__GHOST_URL__` placeholder that corresponds to the configured url that gives a few benefits
- much faster and less memory intensive output transformations through not needing to parse html or markdown - the transform can be achieved using a straightforward regex find+replace
- ability to change to/from or rename subdirectory without any manual updates to the database
- modified existing 4.0 url-transformation migration rather than adding another one and repeating the transformation on posts rows
refs https://github.com/TryGhost/Team/issues/525
We expect the member event tables to be large, and they contain what is
considered metadata. For this reason we do not want to include them in
the export of sites.
refs: https://github.com/TryGhost/Team/issues/510
- In the case that host config is provided, keep staff users within the limiti
- The definition of a staff user is a user with a role other than Contributor, and whose status is not inactive
- Contributors don't count
- Suspended (status inactive) users don't count
- Locked users DO count
- Invited users DO count
- You can't invite more staff users whilst there are pending invites
- You can't unsuspend a user, or change the role on a user in such a way as will take you over your limit
- You can't import staff users - all imported users are automatically set to Contributors
- As part of this work, we are changing the default Ghost user to a Contributor otherwise it uses up a staff user
Note: there is one known active bug with this commit.
- Assume you have one remaining user within your limit. You send an invite, this works.
- You cannot "resend" that invite, it will think you're sending a new invite and hit the limit
- You must "revoke" that invite first, and create a new one
- This bug exists because the resend function uses the add endpoint & does a delete+add, but this hits the permission check before the delete
refs https://github.com/TryGhost/Team/issues/513
refs https://github.com/TryGhost/Team/issues/477
- We have skipped work to improve the 'defaultTo' value when working on Ghost 4.0 release, so adding this comment while context loaded
- defaultTo should not be set to anything as it leads to more maintenance work during major version bump
- having validation might make sense but could lead to similar maintenance work unless it's linked to some global nosion of "supported API versions" used everywhere
no-issue
The slack setting in 1.x did not have a `username` property. When
updating from 1.x, the migration to move the slack setting from a JSON
blob to two individual settings, would assume that the setting in the
database would have a value for the `username` property. This resulted
in errors on SQLite "sqlite does not support inserting default values."
The fix here is to add defaults when reading from the database, meaning
that we will _never_ attempt to insert `undefined`
refs https://github.com/TryGhost/Team/issues/476
* Moved paid subscription events population migration
This migration relies on the members_stripe_customers and
members_stripe_customer_subscriptions tables having no orphaned records
in order for it to correcly generate its data.
The migration to clean up orphaned records in those tables has not been
implemented yet, moving this migration free's up the "14" slot
* Removed orphaned stripe data from SQLite3
SQLite databases do not handle removing orphaned stripe records after a
member has been deleted. Our migration to populate the paid subscription
events relies on each customer and subscription being associated with a
member.
refs https://github.com/TryGhost/Team/issues/494
The migrations in 3.1.0 which added email permissions did not add those
permissions to the roles. This means that whilst we have the permissions
in the database, only the Owner role could use any of them.
This migration ensures that the email related permissions are added to
the correct roles.
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we've introduced backwards-compatible changes to rendering in 3.0 such as srcset and sizes which will only have taken effect on posts created or edited since the changes were made
- 4.0 brings additional changes such as image card width/height
- re-generating the `html` field of all posts from the `mobiledoc` brings all content up to latest rendering output
refs https://github.com/TryGhost/Team/issues/221#issuecomment-759105424
- Mailgun responds to an email send with a provider id in the format `<x@y.com>` but everywhere else it's used in their API it uses the format `x@y.com`
- updates email batch save to strip the brackets, and migration removes brackets from existing records so we no longer have to add special handling for the stored id any time we use it
refs https://github.com/TryGhost/Ghost/issues/12496
- having the logging require here means that workers wanting to use the db are unable to do so without requiring logging as a side-effect
- `connection.loggingHook` does not appear to be widely used for anything outside of specific debugging scenarios when using MySQL so it should be safe to disable until a proper fix is found for workers+logging leaking file descriptors
refs https://github.com/TryGhost/Ghost/pull/12598
- This changeset adds idepmotence to situations where unique contraint has to be dropped or added to the table
- Note '4.0/07-alter-unique-constraint-for-posts-slug.js` was migration that was effected by lack of idempotence
refs https://github.com/TryGhost/Ghost/issues/10318
- Storing JSON object in settings has caused multiple bugs in the past and was considered an antipattern
- This is a last bit in the long process of getting rid of "object" settings in the database. At this point there should be no settings with this type. Yey!
- This commit removes the old boot process and any files that are no longer needed as a result
- Remove the duplicate event for triggering inactive themes to load
- Tidied up a few other bits
no issue
- all migrations should contain a logging message
- info for successful things, or warn if we deviated from what was
expected for some reason
- also added some spacing to make them easier to read
refs https://github.com/TryGhost/Ghost/issues/10318
- API changes introduced:
canary/v4 Admin API
GET /settings/ (browse)
+ "unsplash" present in response as boolean value
GET /settings/:settingName (read)
+ "unsplash" present in response as boolean value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts ONLY boolean format
v3 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts either boolean or object formats
v2 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts object format
refs https://github.com/TryGhost/Ghost/pull/12631
- Initialy writen migration would break if executed twice
- Also accounted for situation if only part of the migration executed leaving only one of the values migrated
refs https://github.com/TryGhost/Team/issues/332
- The last value that has been used in the code was "members"
- By default members will be always "on" starting Ghost 4.0, so there's no need for this flag anymore
- Therefore there's no real need to keep "labs" around
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we added robust url transformation in Ghost 3.0 which ensures newly created or edited content is stored with relative URLs in the database
- this migration loops over all posts (and posts_meta) in the database performing the absolute->relative transformation to normalise data for old posts
refs https://github.com/TryGhost/Ghost/issues/12646
- adds `ghostVersion` property to mobiledoc of all posts so that we can pin the rendering output of existing content to protect against future breaking changes
- cleans any deprecated markdown card names from old Ghost 1.0 content
- cleans up HR cards that had an [accidental payload added](610d801bcf)
refs https://github.com/TryGhost/Ghost/issues/12646
- if the version is missing then the content was created before Ghost 4.0
- setting the version to `'3.0'` means it will continue to use the same rendering output so there are no unexpected breaking changes when migrating content
refs https://github.com/TryGhost/Ghost/issues/12602
* Updated members_status_events table
By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range
* Updated MEGA to handle addition of 'comped' status
With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"
* Updated customQuery for MemberStatusEvent
With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.
* Populated members_status_events table
As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.
The assumptions used in backfilling is that a Member's current status,
is their only status.
refs https://github.com/TryGhost/Ghost/issues/12602
As part of collecting Member event data, we have added a third status
for members "comped" - this fixes the population of the column to handle
this
refs https://github.com/TryGhost/Ghost/issues/12646
- `card-markdown` support is being removed from the default mobiledoc cards so we need to make sure imported content doesn't fail
- removed long-unused `cardName` payload property from tests
refs https://github.com/TryGhost/Ghost/issues/10318
- Object format used in previous "slack" setting was considered an
anti-pattern. Flag structure of separate slack_url and slack_username
values was extracted out of the "slack" JSON.
refs https://github.com/TryGhost/Ghost/issues/12602
As we only want to add events for when an email is changed, we need to store the previous email if we want a complete log of all emails for a member
refs https://github.com/TryGhost/Ghost/issues/12602
* Added members_payment_events table
This table will store successful and unsuccessful payment attempts, and
can be used to calculate gross volume over time.
* Added members_login_events table
This table can be used to audit member logins
* Added members_email_change_events table
This table will allow us to store a history of email addresses associated with a member
* Added members_status_events table
This table will allow us to track the change in status over time for members, as well
as calculate aggregates over time, e.g. paid members over time
* Added members_paid_subscription_events
This table will allow us to track subscriptions changes for members, as well as
calculating MRR over time
refs https://github.com/TryGhost/Ghost/issues/12567
- Changing unique constraint from slug to slug+type should allow for posts and pages to be created with the same slug
- The constraint will be present on application layer for API v4 while we figure out how to deal with it in API v5
Background:
- Ghosts existing boot process is split across multiple files, has affordances for outdated ways of running Ghost and is generally non-linear making it nigh-impossible to follow
- The web of dependencies that are loaded on boot are also impossible to unpick, which makes it really hard to decouple Ghost
- With 4.0 we want to introduce a new, linear, simpler, clearer way to boot up Ghost to unlock decoupling Ghost into much smaller pieces
This commit:
- adds a new ghost.js file which switches between boot mode with `node index` or `node index old` so that if we find bugs we can work around them this week
- Note: the old boot process will go away very soon, but ghost.js will remain as the interface between the command to start Ghost and the application code
- reworks the database migration process into a standalone utility, so that the DB is handled as one simple step of the boot process, decoupled from everything else
- is missing tests for this new db utility
- leaves a lot of work to do around loading core code, services, express apps in a sensible order, as work to fix this would start to break the old boot process
- doesn't use the new maintenance app because we aren't restarting the server here, instead we have the concept of a "core app" that starts in maintenance mode - need to think about how apps will be decoupled in the near future
refs https://github.com/TryGhost/Ghost/issues/12602
This allows us to backfill the data for testing purposes as well as for
use by existing sites. The mechanism for determining the date of an
unsubscribe event is crude, we use the date of the last update, which we
might expect to be _sometimes_ correct, but not always. We store the
`source` as NULL for unsubscribes as we cannot determine that with any
certainty whatsoever.
refs https://github.com/TryGhost/Ghost/issues/12567
- The method was using unneeded lodash dependency and was too complex for what it was doing
- Reshuffled internal code to use native JS filter/forEach iterators
refs https://github.com/TryGhost/Ghost/issues/12567
- Introduced here @@UNIQUE_CONSTRAINTS@@ notation allows to create unique contraints over multiple database fields. This will be needed to change posts' table unique constraint from `slug` to `slug+type`.
- The notation is equivalent to SQL's: UNIQUE(column_name1,column_name2)
- Example use in schema:
posts: {
slug: {type: 'string', maxlength: 191, nullable: false},
type: {type: 'string', maxlength: 50, nullable: false, defaultTo: 'post', validations: {isIn: [['post', 'page']]}},
'@@UNIQUES@@': [
['slug', 'type']
]
}
refs https://github.com/TryGhost/Ghost/issues/12565
- Dropping tables happens sporadicaly, usually during major version releases. It made sense to create a utility based on previous migrations of this type (e.g.: 3.0/03-drop-client-auth-tables.js migration) to avoid code duplication in the future
no-issue
This value is nullable from the Stripe API so we're making sure that we
can store exact values locally.
SQLite3 does not supports altering columns so instead we have to:
1. Create a temp table and copy the data to it
2. Delete the original table
3. Recreate the original table with the necessary modifications
4. Copy the data from the temp table
5. Drop the temp table
refs #12160
This flag will allow us easier filtering of members via the API
* Added status column to members table
This flag will be used to determine if a member is free or paid, rather
than relying on joins with the customers and subscriptions tables.
* Added migration to populate members.status
As we add the column with a default value of "free" we only need to care
about the paid members here. We also preemptively handle migrations for
SQLite where there are > 998 paid members.
closes https://github.com/TryGhost/Ghost/issues/12449
We’re starting to bump into errors with our current exporter due to the size of some of the tables in the db and hitting an issue with Ghost running out of memory during export. The intention for the export/import is not to be backup/restore functionality, but for exporting content and authors.
In addition, exporting and re-importing Stripe secret/publishable keys can cause unexpected side-effects for sites that can has major side-effects. This change -
- Removes `email_batches` and `email_recipients` tables from export data to reduce export size due to large amount of analytics data
- Removes stripe's secret/publishable/webhook keys to avoid unexpected issues with re-import
no refs
Adds new FirstPromoter settings similar to amp, which allows sites to take advantage of FirstPromoter to launch their own member referral program natively.
- Adds new firstpromoter settings group
- Adds `firstpromoter` setting to group
- Adds `firstpromoter_id` setting to group for FirstPromoter referral tracking id
- Updated tests
closes#12181
refs https://github.com/TryGhost/Ghost/pull/12265
- The 'core' value is invalid for settings 'type' column
- The 'core' default value for 'type' column in model is also invalid
- Both need to be removed as they are never used and only introduce confusion into the codebase
refs https://github.com/TryGhost/Ghost/issues/12461
- adds `members.email_count` and `members.email_opened_count` columns to contain cached counts for faster queries when outputting member data via API
- adds migration to populate cached counts with existing data
- tested locally on ~50k members which took ~4sec on mysql
- updates members output serializer to include the new fields in API output
refs https://github.com/TryGhost/Ghost/issues/12421
- nullable so we can distinguish between members that have and haven't received any trackable emails
- indexed because we'll be using this column for sorting
no issue
- cleans up unused tables `emails.{meta,stats}`
- adds timestamp columns `email_recipients.{delivered_at,opened_at,failed_at}` that can be used for event timelines and basic stats aggregation
- indexed because we want to sort by these columns to find the "latest event" when limiting Mailgun events API requests
- adds aggregated stats columns `emails.{delivered_count,opened_count,failed_count}`
- adds a composite index on `email_recipients.[email_id,member_email]` to dramatically speed up `email_recipient` update queries when processing events
- modifies the db initialisation to support an `'@@INDEXES@@'` key in table schema definition for composite indexes
