Commit Graph

4654 Commits

Author SHA1 Message Date
Fabien O'Carroll
9a142cd906 Added aggregateSubscriptionDeltas to permitted options
no-issue

This was missing from https://github.com/TryGhost/Ghost/commit/1dc6fdc.
Without it the option will never be passed into the customQuery method.
2021-02-10 10:06:06 +00:00
Naz
864fcd8a0c Removed settings populateDefaults pre-migration
refs https://github.com/TryGhost/Ghost/issues/12568
refs 5fbc40430b

- Having populateDefaults run before migrations creates a chicken and egg problem where populate defaults can create records that are "non-migratable" as happened in https://github.com/TryGhost/Ghost/issues/12026
2021-02-10 10:34:08 +13:00
Sanne de Vries
67b1745bcb Fixed bug for newsletter bookmarks without author displaying redundant separator 2021-02-09 09:35:45 +01:00
Naz
e0178099a2 🏗 Altered slug unique constraint in posts table
refs https://github.com/TryGhost/Ghost/issues/12567

- Changing unique constraint from slug to slug+type should allow for posts and pages to be created with the same slug
- The constraint will be present on application layer for API v4 while we figure out how to deal with it in API v5
2021-02-09 13:31:12 +13:00
Sanne de Vries
118ceae29c Fixed gmail bug for newsletter bookmark cards not displaying correctly 2021-02-08 16:49:13 +01:00
Hannah Wolfe
0b79abf5b2 Added new, simpler, linear boot process
Background:
- Ghosts existing boot process is split across multiple files, has affordances for outdated ways of running Ghost and is generally non-linear making it nigh-impossible to follow
- The web of dependencies that are loaded on boot are also impossible to unpick, which makes it really hard to decouple Ghost
- With 4.0 we want to introduce a new, linear, simpler, clearer way to boot up Ghost to unlock decoupling Ghost into much smaller pieces

This commit:
- adds a new ghost.js file which switches between boot mode with `node index` or `node index old` so that if we find bugs we can work around them this week
   - Note: the old boot process will go away very soon, but ghost.js will remain as the interface between the command to start Ghost and the application code
- reworks the database migration process into a standalone utility, so that the DB is handled as one simple step of the boot process, decoupled from everything else
- is missing tests for this new db utility
- leaves a lot of work to do around loading core code, services, express apps in a sensible order, as work to fix this would start to break the old boot process
- doesn't use the new maintenance app because we aren't restarting the server here, instead we have the concept of a "core app" that starts in maintenance mode - need to think about how apps will be decoupled in the near future
2021-02-08 11:56:44 +00:00
Fabien O'Carroll
93b1035df5 🐛 Fixed Member model removing labels when unset
closes https://github.com/TryGhost/Ghost/issues/12600

The bookshelf-relations plugin which we use will **remove** all
relations when they are set to an empty array, but will leave them alone
if it's set to undefined.

Our logic to deduplicate uppercase & lowercase version of the same label
was in advertently always setting the labels to an array, but when the
model was saved without passing the labels, this array would be empty.

Here we've added a check which will skip all label handling, if there
are no labels set.
2021-02-08 09:39:46 +00:00
Naz
47843bbfe9 Refactored migration to use transactional utility 2021-02-08 12:25:52 +13:00
Naz
d254a697b9 🏗 Cleanedup app-related settings
refs https://github.com/TryGhost/Ghost/issues/12565
refs f74d459dad

- These fields should have been cleaned up as a result of changed done in referenced commit
2021-02-08 12:25:52 +13:00
Naz
7e7a94def3 Added a note to locale columns
refs https://github.com/TryGhost/Ghost/pull/12599

- These columns are reserved for future features and should not be dropped.
2021-02-08 12:09:23 +13:00
Hannah Wolfe
2b3c00ac32 Changed exit codes to be > 0
ref: https://tldp.org/LDP/abs/html/exitcodes.html

- exit codes being -1 is a mistake that got perpetuated, should be 1 or greater
- for now, just use 1
2021-02-05 20:13:50 +00:00
Fabien O'Carroll
1dc6fdcd66 Added customQuery to handle subscribed aggregates
refs https://github.com/TryGhost/Ghost/issues/12602

In order to build up a list of un/subscribes over time we have to use
the customQuery functionality to run SQL aggregates - this is "hidden"
behind an option, so that we can find{All,Page} as usual.
2021-02-05 16:02:08 +00:00
Fabien O'Carroll
3dbc7ef5b4 Passed MemberSubscribeEvent to members-api
refs https://github.com/TryGhost/Ghost/issues/12602

This allows the Members module to create MemberSubscribeEvents when
appropriate
2021-02-05 16:02:08 +00:00
Fabien O'Carroll
5aab41931f Added skeleton MemberSubscribeEvent model
refs https://github.com/TryGhost/Ghost/issues/12602

Member Events in general are read-only after they've been created, so
we've explicitly disallowed destroying and editing via the model
2021-02-05 16:02:08 +00:00
Fabien O'Carroll
065bbaeafa Populated members_subscribe_events table
refs https://github.com/TryGhost/Ghost/issues/12602

This allows us to backfill the data for testing purposes as well as for
use by existing sites. The mechanism for determining the date of an
unsubscribe event is crude, we use the date of the last update, which we
might expect to be _sometimes_ correct, but not always. We store the
`source` as NULL for unsubscribes as we cannot determine that with any
certainty whatsoever.
2021-02-05 16:02:08 +00:00
Fabien O'Carroll
e39644bd2d Added members_subscribe_events table
refs https://github.com/TryGhost/Ghost/issues/12602
2021-02-05 16:02:08 +00:00
Fabien O'Carroll
873d21d975 Updated addTable util to take optional tableSpec
no-issue

Migrations should contain all the necessary information rather than
relying on external data
2021-02-05 16:02:08 +00:00
Hannah Wolfe
256b77b0e4 Moved sqlite3 override rules to db connection lib
- We have an already established location for special rules based on db client
- Removes the need to require config inside of overrides
2021-02-05 11:51:56 +00:00
Hannah Wolfe
f6a7abdf5a Removed oembed-parser from overrides
- oembed-parser no longer depends on promise-wtf (it's now an archived project)
- therefore the reason for having this here is goooooone
2021-02-05 11:51:51 +00:00
Naz
4b74230624 Refactored createTable migration utility
refs https://github.com/TryGhost/Ghost/issues/12567

- The method was using unneeded lodash dependency and was too complex for what it was doing
- Reshuffled internal code to use native JS filter/forEach iterators
2021-02-04 13:37:20 +13:00
Naz
6b61bcf123 Added unique constraint notation to schema definition
refs https://github.com/TryGhost/Ghost/issues/12567

- Introduced here @@UNIQUE_CONSTRAINTS@@ notation allows to create unique contraints over multiple database fields. This will be needed to change posts' table unique constraint from `slug` to `slug+type`.
- The notation is equivalent to SQL's: UNIQUE(column_name1,column_name2)
- Example use in schema:
posts: {
    slug: {type: 'string', maxlength: 191, nullable: false},
    type: {type: 'string', maxlength: 50, nullable: false, defaultTo: 'post', validations: {isIn: [['post', 'page']]}},
    '@@UNIQUES@@': [
        ['slug', 'type']
    ]
}
2021-02-04 13:37:20 +13:00
Kevin Ansfield
7195a904ba Fixed settings validation error not distinguishing between publishable/secret keys
no issue

- when saving Stripe keys with `stripeDirect: true` config, if either key didn't match the key format the returned validation error always contained `stripe_secret_key`
- updated to output `stripe_publishable_key` if it was the publishable key that was invalid
2021-02-03 16:42:51 +00:00
Thibaut Patel
7d4596adf6 Migrate the default api version from v2 to v3
issue https://github.com/TryGhost/Team/issues/221
2021-02-03 15:02:51 +01:00
Naz
606212d187 🏗 Dropped apps related tables
refs https://github.com/TryGhost/Ghost/issues/12565

- Dropped apps_* tables which were not used since the concept of "apps" has been abandoned
- Uses newly introduced "dropTables" migration utility
- Review for these changes has been completed in
(https://github.com/TryGhost/Ghost/pull/12592)
2021-02-03 12:20:08 +13:00
Daniel Lockyer
0958654457
Updated migration warn message
no issue

- fixed typo and improved message formatting
2021-02-02 16:12:53 +00:00
Naz
96f6ca6218
Fixed drop tables migration utility
refs b5b65b3a60

- The utility was meant to be "irreversible" instead of "transactional"
2021-02-02 16:12:53 +00:00
Naz
23c5ca0795
Added documentation and JSDoc to migration commands
refs https://github.com/TryGhost/Ghost/issues/12567

- Functions were missing documentation and were able to perform operations on multiple columns instead of single one after checking the knex docs (http://knexjs.org/#Schema-unique)
2021-02-02 16:12:53 +00:00
Naz
bf1fcd25ce
Added migration utils for dropping tables
refs https://github.com/TryGhost/Ghost/issues/12565

- Dropping tables happens sporadicaly, usually during major version releases. It made sense to create a utility based on previous migrations of this type (e.g.:  3.0/03-drop-client-auth-tables.js migration) to avoid code duplication in the future
2021-02-02 16:12:53 +00:00
Fabien 'egg' O'Carroll
73f6fd8c51
Enabled Members for all sites (#12582)
no-issue

This removes all references to the members labs setting, any code that was run conditionally behind this flag now runs unconditionally. 

* Removed usage of Members labs flag
* Removed tests for Members disabled
* Added dynamic keypair generation for when setting is missing
2021-02-02 16:12:53 +00:00
Rishabh Garg
26ee648397
🏗 Flattened members subscriptions data in API (#12581)
no refs

- Updates member model serializer to directly set subscriptions on member object instead of `stripe.subscriptions`
- Updates all references to members subscriptions from nested `stripe.subscriptions` to `subscriptions`
- Updates v3 API serializer to still use `stripe.subscriptions`
- Updates tests
2021-02-02 16:12:53 +00:00
Fabien 'egg' O'Carroll
6af2706f10
Updated Admin API and Mega to use status flag (#12579)
no-issue

* Removed support for paid param from v3 & canary API
* Updated active subscription checks to use status flag
* Updated MEGA to use status filter over paid flag
* Removed support for paid option at model level
* Installed @tryghost/members-api@1.0.0-rc.0
* Updated members fixtures
2021-02-02 16:12:51 +00:00
Fabien 'egg' O'Carroll
229295d671
Exited process when members-api fatally errors (#12489)
closes https://github.com/TryGhost/Ghost/issues/12448

This adds handling to force the process to exit when we receive an error
from the members-api on initialisation.
2021-02-02 16:12:10 +00:00
Fabien O'Carroll
5d122c3c62
Updated plan_nickname column to be nullable
no-issue

This value is nullable from the Stripe API so we're making sure that we
can store exact values locally.

SQLite3 does not supports altering columns so instead we have to:
1. Create a temp table and copy the data to it
2. Delete the original table
3. Recreate the original table with the necessary modifications
4. Copy the data from the temp table
5. Drop the temp table
2021-02-02 16:12:10 +00:00
Fabien O'Carroll
cc17de16ba
Added ability to specific table definition to createTable
no-issue
2021-02-02 16:12:10 +00:00
Naz
e33599432d
Fixed maintenance app initialization 2021-02-02 16:12:10 +00:00
Naz
9bff415572
Moved maintenance app to separate package
refs #12568

- Made a package out of maintenance app to keep as little code in the repository
2021-02-02 16:12:08 +00:00
Naz
dcb89f2be0
Changed maintenance app debug code 2021-02-02 16:11:35 +00:00
Naz
244e2374d3
Refactored maintenance app to be a class with DI
refs #12568

- DI pattern through constructor allows for very clear separation of concerns and makes it easy to extract the class into a separate module
2021-02-02 16:11:35 +00:00
Naz
7d9f056e9b
Added error handling and response formatting in maintenance 2021-02-02 16:11:35 +00:00
Naz
d60e36ab57
Added "maintenance" app serving 503 page
refs #12568

- When the server is in maintenance mode (e.g. during migration) it needs to serve a 503 page. Previously this role was delegated to the "frontend" which tightly coupled server bootup and frontend.
- With a dedicated HTTP application serving up 503 server is no longer coupled to the frontend during maintenance
2021-02-02 16:11:35 +00:00
Naz
7e28802b1c
Changed express apps to start after migrations
refs #12568

- Allowing express app (frontend) to be booted after the migrations
decouples server maintenance mode from a need to have the "frontend"
loaded beforehand
2021-02-02 16:11:34 +00:00
Fabien 'egg' O'Carroll
a79ed1170d
Added status property to members (#12570)
refs #12160

This flag will allow us easier filtering of members via the API

* Added status column to members table

This flag will be used to determine if a member is free or paid, rather
than relying on joins with the customers and subscriptions tables.

* Added migration to populate members.status

As we add the column with a default value of "free" we only need to care
about the paid members here. We also preemptively handle migrations for
SQLite where there are > 998 paid members.
2021-02-02 16:11:34 +00:00
Thibaut Patel
b724b2be92
Removed the deprecated send_email_when_published field from the API
refs https://github.com/TryGhost/Team/issues/457
2021-02-02 16:11:34 +00:00
Naz
9163c48f4e
Refactored minimal setup for Ghost boot
refs #12568

- Moved duplicated initialization code to make initialization stages clearer
2021-02-02 16:11:34 +00:00
Naz
75cacc460b
Fixed await for db initialization
refs #12568

- Was missed during the refactor in 62c2a1a05d
2021-02-02 16:11:34 +00:00
Naz
815f67492c
Refactored server boot process to use async/await
refs #12568

- The readivility of multiple nested promises is very poor
- Refactored this piece of code before doing a bigger reshuffle of frontend/server boot process
2021-02-02 16:11:34 +00:00
Thibaut Patel
746ab389ad
Removed the "type" filter from the GET settings api
refs https://github.com/TryGhost/Team/issues/454
2021-02-02 16:11:34 +00:00
Rish
c37c66609d
Removed portal-action and requestSrc
no issue

We added `portal-action` and `requestSrc` in 3.x to allow Portal to handle notifications only for auth actions trigged while using it directly, so that existing themes are not affected in any way. Going forward in 4.0, we don't want to have any special handling in backend for Portal but instead expect themes to handle any Portal specific behavior directly.

- Removes setting of `portal-action` for auth actions like signup
- Removes `requestSrc` being passed through to determine portal actions
2021-02-02 16:11:34 +00:00
Rishabh Garg
5417c4d0e1 Removed stripe keys and analytics data from export (#12573)
closes https://github.com/TryGhost/Ghost/issues/12449

We’re starting to bump into errors with our current exporter due to the size of some of the tables in the db and hitting an issue with Ghost running out of memory during export. The intention for the export/import is not to be backup/restore functionality, but for exporting content and authors.

In addition, exporting and re-importing Stripe secret/publishable keys can cause unexpected side-effects for sites that can has major side-effects. This change -

- Removes `email_batches` and `email_recipients` tables from export data to reduce export size due to large amount of analytics data
- Removes stripe's secret/publishable/webhook keys to avoid unexpected issues with re-import
2021-01-28 14:55:54 +05:30
Fabien O'Carroll
8631710b68 Fixed Members importer usage of linkStripeCustomer
no-issue

The method signature was updated in the refactor and this was missed
2021-01-26 21:26:19 +05:30