Commit Graph

973 Commits

Author SHA1 Message Date
Hannah Wolfe
481e2425af Finalised new boot process
- This commit removes the old boot process and any files that are no longer needed as a result
- Remove the duplicate event for triggering inactive themes to load
- Tidied up a few other bits
2021-02-19 13:15:12 +00:00
Daniel Lockyer
a2dd7c7f7d
Unified migration logging styles
no issue

- all migrations should contain a logging message
- info for successful things, or warn if we deviated from what was
  expected for some reason
- also added some spacing to make them easier to read
2021-02-18 17:35:32 +00:00
Daniel Lockyer
bf960bcecb
Fixed typo in migration message
no issue

- simple rewording to clean up the message
2021-02-18 17:05:00 +00:00
Naz
abb8c1df74 Updated unsplash setting migration to be idempotent
refs https://github.com/TryGhost/Ghost/issues/10318

- Previous version of migration was not following the idempotence rule of migrations
2021-02-18 10:55:59 +13:00
Naz
e0cc314f91 Removed duplicate migration
no issue

- The migration didn't want to go away after rebases combined with renames
2021-02-18 10:55:59 +13:00
Fabien O'Carroll
7785fb6dc5 Refactored usage of pre v4 "unsplash" setting format
refs https://github.com/TryGhost/Ghost/issues/10318

- API changes introduced:

canary/v4 Admin API
GET /settings/ (browse)

+ "unsplash" present in response as boolean value

GET /settings/:settingName (read)

+ "unsplash" present in response as boolean value

PUT /settings/ (edit)

+ "unsplash" updates setting, accepts ONLY  boolean format

v3  Admin API
GET /settings/ (browse)

+ "unsplash" present in response with object value

GET /settings/:settingName (read)

+ "unsplash" present in response with object value

PUT /settings/ (edit)

+ "unsplash" updates setting, accepts either boolean or object formats

v2 Admin API
GET /settings/ (browse)

+ "unsplash" present in response with object value

GET /settings/:settingName (read)

+ "unsplash" present in response with object value

PUT /settings/ (edit)

+ "unsplash" updates setting, accepts object format
2021-02-18 10:55:59 +13:00
Fabien O'Carroll
9cb069eb8e 🏗 Updated "unsplash" setting handling to match updated format
refs https://github.com/TryGhost/Ghost/issues/10318

- JSON object format used in previous "unsplash" setting was considered an
anti-pattern. Flat structure was extracted out of the "unsplash.isActive" JSON.
- The naming convention uses `amp` as  a precedent (https://github.com/TryGhost/Team/issues/331#issuecomment-658815017)
2021-02-18 10:55:59 +13:00
Naz
0097f2037c Fixed slack setting migration to follow idempotance rule
refs https://github.com/TryGhost/Ghost/pull/12631

- Initialy writen migration would break if executed twice
- Also accounted for situation if only part of the migration executed leaving only one of the values migrated
2021-02-18 10:55:32 +13:00
Naz
ea6d656457 🏗 Removed labs from setting
refs https://github.com/TryGhost/Team/issues/332

- The last value that has been used in the code was "members"
- By default members will be always "on" starting Ghost 4.0, so there's no need for this flag anymore
- Therefore there's no real need to keep "labs" around
2021-02-17 19:55:16 +13:00
Kevin Ansfield
cabf718ae1
Added migration to transform all post content urls from absolute to relative (#12658)
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221

- we added robust url transformation in Ghost 3.0 which ensures newly created or edited content is stored with relative URLs in the database
- this migration loops over all posts (and posts_meta) in the database performing the absolute->relative transformation to normalise data for old posts
2021-02-16 19:00:04 +00:00
Fabien O'Carroll
037b2ce761 Renamed migration to fill 01
no-issue
2021-02-16 18:19:09 +00:00
Fabien O'Carroll
87471fd5c6 Revert "Updated plan_nickname column to be nullable"
This reverts commit 5d122c3c62.
2021-02-16 18:19:09 +00:00
Kevin Ansfield
3dfbb6296b Fixed duplicate migration number 17 2021-02-16 12:33:24 +00:00
Kevin Ansfield
66fe678cb7
Added 4.0 mobiledoc update migration (#12648)
refs https://github.com/TryGhost/Ghost/issues/12646

- adds `ghostVersion` property to mobiledoc of all posts so that we can pin the rendering output of existing content to protect against future breaking changes
- cleans any deprecated markdown card names from old Ghost 1.0 content
- cleans up HR cards that had an [accidental payload added](610d801bcf)
2021-02-16 12:13:30 +00:00
Kevin Ansfield
2d091fa8f9 Set mobiledoc.ghostVersion: '3.0' if missing on imported content
refs https://github.com/TryGhost/Ghost/issues/12646

- if the version is missing then the content was created before Ghost 4.0
- setting the version to `'3.0'` means it will continue to use the same rendering output so there are no unexpected breaking changes when migrating content
2021-02-16 12:08:40 +00:00
Fabien 'egg' O'Carroll
f4cb5c57c6
Updated members_status_events table (#12647)
refs https://github.com/TryGhost/Ghost/issues/12602

* Updated members_status_events table

By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range

* Updated MEGA to handle addition of 'comped' status

With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"

* Updated customQuery for MemberStatusEvent

With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.

* Populated members_status_events table

As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.

The assumptions used in backfilling is that a Member's current status,
is their only status.
2021-02-16 10:38:36 +00:00
Fabien 'egg' O'Carroll
da9cd3b9d6
Updated status population to handle comped status (#12651)
refs https://github.com/TryGhost/Ghost/issues/12602

As part of collecting Member event data, we have added a third status
for members "comped" - this fixes the population of the column to handle
this
2021-02-16 10:37:06 +00:00
Kevin Ansfield
70486ccb4e
Updated importer to transform 'card-markdown' to 'markdown' (#12650)
refs https://github.com/TryGhost/Ghost/issues/12646

- `card-markdown` support is being removed from the default mobiledoc cards so we need to make sure imported content doesn't fail
- removed long-unused `cardName` payload property from tests
2021-02-15 16:42:01 +00:00
Fabien O'Carroll
8880cc82c7 Refactored usage of the pre v4 "slack" setting
refs https://github.com/TryGhost/Ghost/issues/10318

- Change/compatibility table for this changeset:

Importer

+ "slack_username"  import
- "slack_url" ignore
+ "slack.username" import
- "slack.url" ignore

v3 & canary/v4 Admin APIs
GET /settings/ (browse)

+ "slack_username" present in response
+ "slack_url" present in response
+ "slack" present in response

GET /settings/:settingName (read)

+ "slack_username" present in response
+ "slack_url" present in response
+ "slack" present in response

PUT /settings/ (edit)

+ "slack_username" updates setting
+ "slack_url" updates setting
+ "slack" !NOTE! updates setting unless "slack_username" or "slack_ur"l are set (updated keys take priority)

v2 Admin API
GET /settings/ (browse)

- "slack_username" NOT present in response
- "slack_url" NOT present in response
+ "slack" present in response

GET /settings/:settingName (read)

- "slack_username" ignored (404)
- "slack_url" ignored (404)
- "slack" present in response

PUT /settings/ (edit)

- "slack_username" ignored (no error)
- "slack_url" ignored (no error)
+ "slack" updates setting
2021-02-15 15:25:54 +13:00
Fabien O'Carroll
12c28fe66e 🏗 Updated "slack" setting into "slack_{url,username}"
refs https://github.com/TryGhost/Ghost/issues/10318

- Object format used in previous "slack" setting was considered an
anti-pattern. Flag structure of separate slack_url and slack_username
values was extracted out of the "slack" JSON.
2021-02-15 15:25:54 +13:00
Rishabh Garg
1d281581eb
Updated members_email_change_events to store previous email (#12641)
refs https://github.com/TryGhost/Ghost/issues/12602

As we only want to add events for when an email is changed, we need to store the previous email if we want a complete log of all emails for a member
2021-02-12 16:15:31 +00:00
Thibaut Patel
2f547b2974 Added the frontmatter column to meta
issue https://github.com/TryGhost/Team/issues/452
2021-02-12 15:31:04 +01:00
Fabien O'Carroll
905421b774 Populated members_paid_subscription_events table
refs https://github.com/TryGhost/Ghost/issues/12602

By backfilling the data, we allow exisiting sites to view their MRR over
time
2021-02-12 13:55:14 +00:00
Fabien 'egg' O'Carroll
f745fa7621
Updated members_paid_subscription_events table (#12636)
refs https://github.com/TryGhost/Ghost/issues/12602

This allows us to store the stripe_id of the subscription events
2021-02-11 12:45:18 +00:00
Fabien O'Carroll
5624e14b1b Renamed duplicate migration
no-issue
2021-02-11 11:02:18 +00:00
Fabien 'egg' O'Carroll
601d135b43
Added members event tables (#12618)
refs https://github.com/TryGhost/Ghost/issues/12602

* Added members_payment_events table

This table will store successful and unsuccessful payment attempts, and
can be used to calculate gross volume over time.

* Added members_login_events table

This table can be used to audit member logins

* Added members_email_change_events table

This table will allow us to store a history of email addresses associated with a member

* Added members_status_events table

This table will allow us to track the change in status over time for members, as well
as calculate aggregates over time, e.g. paid members over time

* Added members_paid_subscription_events

This table will allow us to track subscriptions changes for members, as well as 
calculating MRR over time
2021-02-11 10:54:30 +00:00
Naz
864fcd8a0c Removed settings populateDefaults pre-migration
refs https://github.com/TryGhost/Ghost/issues/12568
refs 5fbc40430b

- Having populateDefaults run before migrations creates a chicken and egg problem where populate defaults can create records that are "non-migratable" as happened in https://github.com/TryGhost/Ghost/issues/12026
2021-02-10 10:34:08 +13:00
Naz
e0178099a2 🏗 Altered slug unique constraint in posts table
refs https://github.com/TryGhost/Ghost/issues/12567

- Changing unique constraint from slug to slug+type should allow for posts and pages to be created with the same slug
- The constraint will be present on application layer for API v4 while we figure out how to deal with it in API v5
2021-02-09 13:31:12 +13:00
Hannah Wolfe
0b79abf5b2 Added new, simpler, linear boot process
Background:
- Ghosts existing boot process is split across multiple files, has affordances for outdated ways of running Ghost and is generally non-linear making it nigh-impossible to follow
- The web of dependencies that are loaded on boot are also impossible to unpick, which makes it really hard to decouple Ghost
- With 4.0 we want to introduce a new, linear, simpler, clearer way to boot up Ghost to unlock decoupling Ghost into much smaller pieces

This commit:
- adds a new ghost.js file which switches between boot mode with `node index` or `node index old` so that if we find bugs we can work around them this week
   - Note: the old boot process will go away very soon, but ghost.js will remain as the interface between the command to start Ghost and the application code
- reworks the database migration process into a standalone utility, so that the DB is handled as one simple step of the boot process, decoupled from everything else
- is missing tests for this new db utility
- leaves a lot of work to do around loading core code, services, express apps in a sensible order, as work to fix this would start to break the old boot process
- doesn't use the new maintenance app because we aren't restarting the server here, instead we have the concept of a "core app" that starts in maintenance mode - need to think about how apps will be decoupled in the near future
2021-02-08 11:56:44 +00:00
Naz
47843bbfe9 Refactored migration to use transactional utility 2021-02-08 12:25:52 +13:00
Naz
d254a697b9 🏗 Cleanedup app-related settings
refs https://github.com/TryGhost/Ghost/issues/12565
refs f74d459dad

- These fields should have been cleaned up as a result of changed done in referenced commit
2021-02-08 12:25:52 +13:00
Naz
7e7a94def3 Added a note to locale columns
refs https://github.com/TryGhost/Ghost/pull/12599

- These columns are reserved for future features and should not be dropped.
2021-02-08 12:09:23 +13:00
Fabien O'Carroll
065bbaeafa Populated members_subscribe_events table
refs https://github.com/TryGhost/Ghost/issues/12602

This allows us to backfill the data for testing purposes as well as for
use by existing sites. The mechanism for determining the date of an
unsubscribe event is crude, we use the date of the last update, which we
might expect to be _sometimes_ correct, but not always. We store the
`source` as NULL for unsubscribes as we cannot determine that with any
certainty whatsoever.
2021-02-05 16:02:08 +00:00
Fabien O'Carroll
e39644bd2d Added members_subscribe_events table
refs https://github.com/TryGhost/Ghost/issues/12602
2021-02-05 16:02:08 +00:00
Fabien O'Carroll
873d21d975 Updated addTable util to take optional tableSpec
no-issue

Migrations should contain all the necessary information rather than
relying on external data
2021-02-05 16:02:08 +00:00
Hannah Wolfe
256b77b0e4 Moved sqlite3 override rules to db connection lib
- We have an already established location for special rules based on db client
- Removes the need to require config inside of overrides
2021-02-05 11:51:56 +00:00
Naz
4b74230624 Refactored createTable migration utility
refs https://github.com/TryGhost/Ghost/issues/12567

- The method was using unneeded lodash dependency and was too complex for what it was doing
- Reshuffled internal code to use native JS filter/forEach iterators
2021-02-04 13:37:20 +13:00
Naz
6b61bcf123 Added unique constraint notation to schema definition
refs https://github.com/TryGhost/Ghost/issues/12567

- Introduced here @@UNIQUE_CONSTRAINTS@@ notation allows to create unique contraints over multiple database fields. This will be needed to change posts' table unique constraint from `slug` to `slug+type`.
- The notation is equivalent to SQL's: UNIQUE(column_name1,column_name2)
- Example use in schema:
posts: {
    slug: {type: 'string', maxlength: 191, nullable: false},
    type: {type: 'string', maxlength: 50, nullable: false, defaultTo: 'post', validations: {isIn: [['post', 'page']]}},
    '@@UNIQUES@@': [
        ['slug', 'type']
    ]
}
2021-02-04 13:37:20 +13:00
Naz
606212d187 🏗 Dropped apps related tables
refs https://github.com/TryGhost/Ghost/issues/12565

- Dropped apps_* tables which were not used since the concept of "apps" has been abandoned
- Uses newly introduced "dropTables" migration utility
- Review for these changes has been completed in
(https://github.com/TryGhost/Ghost/pull/12592)
2021-02-03 12:20:08 +13:00
Daniel Lockyer
0958654457
Updated migration warn message
no issue

- fixed typo and improved message formatting
2021-02-02 16:12:53 +00:00
Naz
96f6ca6218
Fixed drop tables migration utility
refs b5b65b3a60

- The utility was meant to be "irreversible" instead of "transactional"
2021-02-02 16:12:53 +00:00
Naz
23c5ca0795
Added documentation and JSDoc to migration commands
refs https://github.com/TryGhost/Ghost/issues/12567

- Functions were missing documentation and were able to perform operations on multiple columns instead of single one after checking the knex docs (http://knexjs.org/#Schema-unique)
2021-02-02 16:12:53 +00:00
Naz
bf1fcd25ce
Added migration utils for dropping tables
refs https://github.com/TryGhost/Ghost/issues/12565

- Dropping tables happens sporadicaly, usually during major version releases. It made sense to create a utility based on previous migrations of this type (e.g.:  3.0/03-drop-client-auth-tables.js migration) to avoid code duplication in the future
2021-02-02 16:12:53 +00:00
Fabien O'Carroll
5d122c3c62
Updated plan_nickname column to be nullable
no-issue

This value is nullable from the Stripe API so we're making sure that we
can store exact values locally.

SQLite3 does not supports altering columns so instead we have to:
1. Create a temp table and copy the data to it
2. Delete the original table
3. Recreate the original table with the necessary modifications
4. Copy the data from the temp table
5. Drop the temp table
2021-02-02 16:12:10 +00:00
Fabien O'Carroll
cc17de16ba
Added ability to specific table definition to createTable
no-issue
2021-02-02 16:12:10 +00:00
Fabien 'egg' O'Carroll
a79ed1170d
Added status property to members (#12570)
refs #12160

This flag will allow us easier filtering of members via the API

* Added status column to members table

This flag will be used to determine if a member is free or paid, rather
than relying on joins with the customers and subscriptions tables.

* Added migration to populate members.status

As we add the column with a default value of "free" we only need to care
about the paid members here. We also preemptively handle migrations for
SQLite where there are > 998 paid members.
2021-02-02 16:11:34 +00:00
Rishabh Garg
5417c4d0e1 Removed stripe keys and analytics data from export (#12573)
closes https://github.com/TryGhost/Ghost/issues/12449

We’re starting to bump into errors with our current exporter due to the size of some of the tables in the db and hitting an issue with Ghost running out of memory during export. The intention for the export/import is not to be backup/restore functionality, but for exporting content and authors.

In addition, exporting and re-importing Stripe secret/publishable keys can cause unexpected side-effects for sites that can has major side-effects. This change -

- Removes `email_batches` and `email_recipients` tables from export data to reduce export size due to large amount of analytics data
- Removes stripe's secret/publishable/webhook keys to avoid unexpected issues with re-import
2021-01-28 14:55:54 +05:30
Aileen Nowak
c9d1fa0b96 Updated links to ghost.org sites
no issue

Follow-up task of the updated Ghost Docs structure. Updated links reflecting the new structure to prevent unnecessary 404s and redirects.
2021-01-20 09:59:45 +13:00
Rishabh Garg
592c1bd46c
Added FirstPromoter Integration (#12540)
no refs

Adds new FirstPromoter settings similar to amp, which allows sites to take advantage of FirstPromoter to launch their own member referral program natively.

- Adds new firstpromoter settings group
- Adds `firstpromoter` setting to group
- Adds `firstpromoter_id` setting to group for FirstPromoter referral tracking id
- Updated tests
2021-01-15 19:05:15 +05:30
Sumedh
e756af65f2 Removed invalid default 'core' value in settings type
closes #12181
refs https://github.com/TryGhost/Ghost/pull/12265

- The 'core' value is invalid for settings 'type' column
- The 'core' default value for 'type' column in model is also invalid
- Both need to be removed as they are never used and only introduce confusion into the codebase
2021-01-04 15:32:57 +13:00