Commit Graph

924 Commits

Author SHA1 Message Date
Naz Gargol
0225936292
Removed subscribers from the codebase (#11153)
refs https://github.com/TryGhost/Ghost/pull/11152

- Added subscribers table drop migration
- Removed subscribers from schema
- Removed subscribers controllers/routes/regression tests
- Removed subscriber related API code
- Removed subscribers from internal apps
- Removed subscriber importer
- Removed subscriber model
- Removed subscriber related permissions
- Removed webhook code related to subscribers
- When upgrading to v3 it is on the site admin to migrate all zapps or any other webhook clients to use members
- Removed subscriber-specific translation
- Removed subscriber lab flag
2019-10-09 11:47:04 +02:00
Fabien O'Carroll
b77026870b Moved migrations from 3.0 to 2.34
no-issue
2019-10-09 16:24:51 +07:00
Fabien O'Carroll
cbb6337ae4 Prefixed stripe_customers_subscriptions with members
no-issue
2019-10-09 16:24:51 +07:00
Fabien O'Carroll
0c32dfaa30 Added migrations for stripe tables
no-issue
2019-10-09 16:24:51 +07:00
Fabien O'Carroll
a4ff87a774 Added stripe subscriptions & updated customers table
no-issue
2019-10-09 16:24:51 +07:00
Rishabh Garg
7dc2eb2a1e
Added new requirePaymentForSignup setting for members (#11214)
* Added new `requirePaymentForSignup` setting for members

no issue

- Adds new `requirePaymentForSignup` setting flag for members, `false` by default.

- Wired members API `allowSelfSignup` to `requirePayment` setting
2019-10-08 22:00:46 +05:30
Kevin Ansfield
6028fde666 Merge branch 'master' into v3 2019-10-08 13:58:08 +01:00
Fabien O'Carroll
f1ef801b78 🐛 Fixed error when during migration to 2.32.0 on mysql (#11208)
closes #11207

MySQL doesn't allow unqiue keys with a length of more than 191 when using InnoDB with utfmb4. These changes will ensure any incorrect tables created are fixed and have the
correct length for customer_id

* Changed `customer_id` to non-unique column
* Nooped the 2.32 `members_stripe_customers` migration
* Added migration to recreate `members_stripe_customers` table
  * sqlite doesn't allow `ALTER TABLE` queries so this is the cleanest solution considering the table is not yet in use
2019-10-08 11:07:29 +01:00
Naz Gargol
e214838039
Improve visibility's default handling (#11183)
no issue

- Moved default_content_visibility out of labs as we should be extra careful with what is exposed in the labs + it doesn't really belong there.
2019-10-02 11:08:10 +02:00
Naz Gargol
ff13821b27
Members post gating options (#11160)
no issue

- Removed uses of `visibility` column in frontend url service configs
- The value of `visibility` is always set to 'public' in posts at the moment and doesn't serve any specific purpose when used with these filters.
- Allowed new visibility attributes in post model
- `posts.visibility` column is being repurposed for the needs of member content gating
- Added test for visibility editing in Admin API
- Corrected test schema checks for Admin API post/page responses
2019-09-26 15:38:35 +02:00
Naz Gargol
a562f09c0d
🏗 Migrated subscribers data to members (#11152)
no issue 

- Populates members table with existing subscribers. Only takes into account columns we know already exist and need to be copied i.e `name`/`email`
2019-09-26 10:39:20 +02:00
Fabien O'Carroll
a62b014905 Renamed members_stripe_info to members_stripe_customers
no-issue

This is more specific and better if we start adding more stripe tables
2019-09-26 12:58:29 +07:00
Fabien O'Carroll
9b3d45d4c4 Corrected number for members name column migration
no-issue
2019-09-26 12:58:29 +07:00
Naz Gargol
d54be917d1
Added name column back to members table (#11151)
refs 294f3769cb

- We have a need for name field now :)
- This time `name` is nullable !
2019-09-25 10:44:50 +02:00
Fabien O'Carroll
fd1db4ffac Added migration for members-stripe-info table
no-issue
2019-09-25 10:05:30 +07:00
Fabien O'Carroll
6859e9a9a1 Added members-stripe-info table
no-issue

This will be used to store stripe specific information for members

customer_id has a max length of 255 https://stripe.com/docs/upgrades

member_id is not unique as we cannot ensure that a member doesn't have
more than one customer object associated with them. e.g. if they signup
twice, or if they cancel, and signup again, creating a new customer.

We probably won't handle this case to begin with, but we will keep the
data intact.
2019-09-25 10:05:30 +07:00
Kevin Ansfield
3eb11533bd Marked the client auth table drop migration as irreversible
no issue

- bumps `knex-migrator` so it supports irreversible migrations
- marks the `03-drop-client-auth` migration as irreversible because it destroys data that is not recoverable and is required for earlier versions of Ghost to function
2019-09-23 17:22:46 +01:00
Kevin Ansfield
fb1b207db9 Fixed error rolling back the "remove empty strings" migration
no issue

- rollbacks have switched to using transactions but the migration code was copied from an old migration coded before that switch
- `down()` is no longer called with an object that contains a `connection` key, it has `transacting` instead
2019-09-18 16:08:18 +01:00
Kevin Ansfield
5be275bd46
Added migration for re-generating html of all posts (#11133)
refs https://github.com/TryGhost/Ghost/issues/10445

- re-generates HTML for every post so that they are using the most recent Koenig renderer output
2019-09-16 17:44:00 +01:00
Kevin Ansfield
ea7a677e4a
Added migration to normalize all empty strings to NULL (#11132)
refs https://github.com/TryGhost/Ghost/issues/10388, original PR https://github.com/TryGhost/Ghost/pull/10428

- re-introduces the migration which normalizes all empty strings in the database to `NULL`
- this was previously reverted due to being too large of a migration for a minor but a major is a good time to do it
2019-09-16 17:43:48 +01:00
Kevin Ansfield
39db5bd177 Prefixed 3.0 migrations with 0 to preserve order
no issue

- `knex-migrator` will run migrations in the order that nodejs provides when running `fs.readDirSync` which in most cases is strict alphabetical
- 3.0 will shortly have more than 9 migrations which was resulting in the migration order being 1, 10, 2
- prefixing all single-digit migrations with `0` means that strict alphabetical ordering results in the expected order
2019-09-16 13:21:39 +01:00
Kevin Ansfield
06365b282f Fixed migrations folder name
no issue

- 3.0.0 was renamed to 3.0 for consistency
2019-09-16 11:54:03 +01:00
Kevin Ansfield
e57e19ec31
🏗 Migrated posts.page column to posts.type (#11111)
refs https://github.com/TryGhost/Ghost/issues/10922

- adds migrations to...
  1. add `post.type` column
  2. populate `post.type` column based on `post.page` value
  3. drop `post.page` column
- updates all code paths to work with `post.type` in place of `post.page`
- adds `nql-map-key-values` transformer for mapping `page`->`type` in `filter` params when using the v2 API
- modifies importer to handle `post.page`->`post.type` transformation when importing older export files
2019-09-16 11:51:54 +01:00
Rish
d2bd3975f9 Fixed migration order 2019-09-16 14:22:01 +05:30
Rish
0b34b508d8 Fixed migration order and folder naming 2019-09-16 14:19:13 +05:30
Rishabh Garg
8ec12d9eee
🏗 Extracted post metadata in new post_meta table (#11102)
NOTE: The post metadata table split is purely an internal optimization for v3 and doesn't require or expect any external actions including related API usage in v3

We keep running into issues adding new fields to the post table because there are too many fields making the post table "too wide". We have also hit MySQL limitations in how many bytes can be in a row (64kb) with post table.

In v3, we decided to split the 8 post fields (meta, twitter and og) used for meta data into a posts_meta table as these 8 fields are all "problem" `varchar` fields and make sense logically grouped together. The API layer is unaffected by the split as input/output serializers ensure the data flow works the same way as it was in v2. Only thing to note is json export in v3 will have slightly different structure with posts meta fields as separate.

- Creates new post_meta schema/table with 8 fields (2 meta_* , 3 twitter_* and 3 og_*)
- Update relations between post and post_meta table
- Update input/output serializers to keep existing API behavior
- Avoids new entry in post_meta table for post where all meta fields are null
- Keeps the current fields API param behavior
- Handles migration of existing posts to new table structure
- Updates importer/exporter to work seamlessly with table changes
2019-09-16 14:15:55 +05:30
Rish
caa121cae5 Fixed log usage for columnMigration command 2019-09-13 19:54:18 +05:30
Kevin Ansfield
1e905e1ef6
Added migrations to drop v0.1 auth tables (#11108)
no issue

- drops now-unused `accesstokens`, `refreshtokens`, `clients`, and `client_trusted_domains` tables
- no rollback because the db schema for the tables no longer exists
2019-09-13 13:36:20 +01:00
Kevin Ansfield
d645afb416 Removed all accesstokens and refreshtokens related code
no issue

- v0.1 is ☠️ so the access/refresh token based auth is no longer used
- removed all code related to the `accesstokens` and `refreshtokens` tables
- removed all `passport` related dependencies as it's no longer used
2019-09-12 15:45:27 +01:00
Kevin Ansfield
33fe21f888 Removed all clients and client_trusted_domains related code
no issue

- v0.1 is ☠️ so there's no longer any use of client auth
- removes all code related to `clients` and `client_trusted_domains`
- noops the "add backup client" migration in 1.7 because the referenced fixture no longer exists causing migrations and consequently all regression tests to fail
2019-09-12 15:09:49 +01:00
Kevin Ansfield
1752132051 Merge branch 'master' into v3 2019-09-10 09:50:15 +01:00
Fabien O'Carroll
294f3769cb Removed name and password columns from members table
no-issue

We have no need for these right now and it is easier to drops the
columns, rather than to modify the name column to nullable
2019-09-05 11:14:50 +08:00
Fabien O'Carroll
7382967613 Added createColumnMigration helper
no-issue
2019-09-05 11:14:50 +08:00
Rishabh Garg
b875cc339d
🔥 Dropped unused ghost_auth_* user fields
no issue

- Drops `ghost_auth_access_token` and `ghost_auth_id` fields since not used anymore
- Adds migration for dropping these columns from users table
- Drops Auth strategy - `ghostStrategy` - since its not used anymore
2019-09-03 20:48:42 +05:30
Kevin Ansfield
f038fbe0aa Fixed "Unable to update nested relation" errors when rolling back migrations
no issue

- we recently started wrapping rollbacks in transactions (https://github.com/TryGhost/knex-migrator/pull/161)
- in a number of migrations we were calling `model.destroy()` without passing through the options which includes the current transaction
- for models which are using `bookshelf-relations` this could result in an internal `SQLITE_BUSY: database is locked` error because it tries to run queries against tables that have been locked by previous queries in the transaction
- by passing through the options when calling `.destroy()` it allows the `bookshelf-relations` to re-use the same transction avoiding the database lock problems
2019-08-20 11:20:52 +01:00
Kevin Ansfield
80fbfd7a85 Revert post.page->post.type handling
no issue
- the column addition/removal can be too slow for large sites
- will be added back in 3.0

---

Revert "Fixed canary api for page/type column"
This reverts commit a5a7e7e919.

Revert "Updated frontend canary url config for page/type"
This reverts commit 19100ec5e6.

Revert "Updated canary api to handle type column correctly (#11006)"
This reverts commit c3e8ba0523.

Revert "Ensured `page` filter works in routes.yaml"
This reverts commit 9037c19e50.

Revert "Replaced usage of mongo util with nql-map-key-values"
This reverts commit 8c5f1d0ef0.

Revert "Added shared nql-map-key-values module"
This reverts commit ef4fd4b8ef.

Revert "Ensured page prop is present on content api response"
This reverts commit cfa0a0862b.

Revert "Fixed failing regression tests"
This reverts commit 9c2bb3811f.

Revert "Updated xmlrpc and slack service to use type column"
This reverts commit 44a02c7d36.

Revert "Updated v0.1 posts api to work with type column"
This reverts commit 2c81d7c914.

Revert "Removed updates to v0.1 specific code"
This reverts commit 08d83c1f53.

Revert "Added missing context from ValidationError"
This reverts commit cd45ab4f54.

Revert "Renamed page->type in the page&posts serializers"
This reverts commit df99e724e3.

Revert "Added mongo helper to input serializers"
This reverts commit fb8eadb4a8.

Revert "Passed mongoTransformer through to NQL"
This reverts commit 0ae3f0fdfc.

Revert "Permitted mongoTransformer option for read methods"
This reverts commit a89376bf26.

Revert "Updated the count plugin to reference the type column"
This reverts commit a52f15d3d3.

Revert "Updated hashes for db integrity check"
This reverts commit bb6b337be3.

Revert "Remove page column and remaining references"
This reverts commit 9d7190d692.

Revert "Added type column to data generator"
This reverts commit e59806cb45.

Revert "Removed references to page column in rss tests"
This reverts commit 04d0f855de.

Revert "Removed page column references in validation tests"
This reverts commit f0afbc5cc0.

Revert "Updated the post model to use the `type` column"
This reverts commit 1189bc823a.

Revert "Updated url service to use type column"
This reverts commit 61612ba8fd.

Revert "Updated the v2 api to deal with type column"
This reverts commit 57afb2de2b.

Revert "Added type property to post model defaults"
This reverts commit dc3345b1c5.

Revert "Added type property to the default post fixtures"
This reverts commit 82d8c38033.

Revert "Added type column to posts table"
This reverts commit 9b85fc6a69.
2019-08-19 12:02:14 +01:00
Kevin Ansfield
3764e023fb No-op'd post.page->post.type migrations, added cleanup post.type->post.page migrations 2019-08-19 12:02:14 +01:00
Kevin Ansfield
25f91aa102
Fixed importer not handling post.page->post.type conversion (#11025)
no issue

- updates the attribute sanitiser of the posts importer to convert `post.page=true/false` to `post.type='page'/'post'
- gives precedence to `post.type` if an imported post somehow has both `post.page` and `post.type` attributes
2019-08-16 19:41:38 +01:00
Kevin Ansfield
ff1ac49b0a
Excluded mobiledoc_revisions table from exports (#10993)
no issue

- the `mobiledoc_revisions` table can grow very large in certain circumstances which can result in Out-Of-Memory errors when performing backups, resulting in failed upgrades
- adds `mobiledoc_revisions` to the exporter excluded tables list as a temporary solution until we have safer export creation and/or improved revision handling
2019-08-12 11:33:16 +01:00
Nazar Gargol
3bd3570592 🐛 Fixed duplication error when importing posts without slugs
refs #8717

-  The posts without slugs should not be taken into account when detecting duplicates as slug field is not required when importing.
- Ideal solution would require generating slug before duplicate detection phase. This would cause duplicate detection to take 'title' into account which didn't happen before.
2019-08-12 09:20:31 +01:00
Fabien O'Carroll
acd1a7fd69 Fixed naming of backup cotent perm migration
no-issue

This file got accidentally renamed in 532fdb0806
2019-08-09 15:00:26 +08:00
Fabien O'Carroll
b6f2bc33b7 Moved page/type column migrations to correct version
refs #10922
2019-08-09 14:53:40 +08:00
Fabien O'Carroll
b45e955dea Ensured that the page column can be rolled back
refs #10922

When rolling back the removal of the page column, we must re-add it, but
the definition for it has been removed from the schema, so we must
hardcode the definition.
2019-08-09 14:39:01 +08:00
Fabien O'Carroll
1096dc2085 Added ability to pass columnSpec to addTableColumn
refs #10922

This gives us the ability to add columns that have since been removed
from the schema, for example in a down migration.
2019-08-09 14:39:01 +08:00
Fabien O'Carroll
cd45ab4f54 Added missing context from ValidationError
refs #10922

This is necessary to keep existing functionality in v0.1
2019-08-09 14:39:01 +08:00
Fabien O'Carroll
51fb0815b4 Created migrations for page -> type column
refs #10822
2019-08-09 14:39:01 +08:00
Fabien O'Carroll
9d7190d692 Remove page column and remaining references
refs #10922
2019-08-09 14:39:01 +08:00
Fabien O'Carroll
82d8c38033 Added type property to the default post fixtures
refs #10922
2019-08-09 14:39:01 +08:00
Fabien O'Carroll
9b85fc6a69 Added type column to posts table
refs #10922

This column is limited to the two values 'post' and 'page'
2019-08-09 14:39:01 +08:00
Nazar Gargol
6779732cd0 Fixed regression tests for v2/admin/db
no-issue
2019-08-09 10:57:49 +08:00
Nazar Gargol
532fdb0806 Added migrations for scheduler integration, permission and role
refs #10060

- Modification of https://github.com/TryGhost/Ghost/pull/10974/files
- Added publish permission migrations for all roles having "post": "all" permission
2019-08-07 14:56:51 +02:00
Nazar Gargol
00f95e7328 Migrated schedules controller to v2
closes #10060

- Implemented scheduling for posts and pages
- Added cache invalidation when scheduling
- Refactored admin token eneration function to accept existing key as parameter in tests
- Added Ghost Scheduler Integration fixture
- Added fixture for permissions for post publish action
- Migrated getScheduled method to v2
- Did not add support for 'from' and 'to' parameters as they were not used by DefaultScheduler
- This method needs rethinking in a long run as it's an ugly hack and should rather become proper endpoint that returns JSON data instead of models
- Removed unused auth middleware from v2 routes
- Added internal scheduler role
- Implemetnted transactions in v2 frame
- This takes into account scenario mentioned in c93f03b87e
- Specifically:
>if two queries happening in a transaction we have to signalise
  knex/mysql that we select for an update
  otherwise the following case happens:
  you fetch posts for an update
  a user requests comes in and updates the post (e.g. sets title to "X")
  you update the fetched posts, title would get overriden to the old one
2019-08-07 14:51:36 +02:00
Fabien O'Carroll
42c9904a8f Added migrations for backupContent permission
no-issue
2019-08-07 17:57:26 +08:00
Fabien O'Carroll
910e15e643 Added backupContent permission fixture
no-issue

This permission was overlooked during the creation of the ghost-backup
integration.
2019-08-07 17:57:26 +08:00
Fabien O'Carroll
373627223c
🐛 Ensured import does not override private setting (#10882)
closes #10788

This adds an extra filter to the preImport method of the settings
importer to removes settings with the key `is_private`

This message is specifically only for when an import has privacy mode ON
and the current site has privacy mode OFF.
2019-08-06 16:15:40 +08:00
Kevin Ansfield
d96be4907e
Fixed relative canonical_url values not being stored as root-relative (#10989)
no issue

- we try to store all urls as relative paths where possible in Ghost so that the `config.url` value can be changed
- all relative paths are stored as root-relative except for the `post.canonical_url` field which was storing subdirectory-relative paths
- adds a migration to put the subdirectory prefix onto any relative canonical_url paths
- updates the canonical_url input serialiser to keep the subdirectory rather than stripping it to match all other url fields
2019-08-05 13:56:28 +01:00
Fabien O'Carroll
21427ad73f
Created DB Backup integration (#10974)
* Simplified db controller permissions options

The existing objects were confusing because they did the same thing as
setting permissions to true, but gave the impressions that something
special was happening/required.

* Added DB Backup Integration Role

This will allow us to assign certain api_keys this role, in order to
automate db backups

* Allowed admin api_keys to have configurable roles

This will allow keys for the admin api to do customised things such as db export

* Added ghost-backup integration to fixtures

* Added migrations for DB Backup Integration and role
2019-08-02 17:28:02 +08:00
Aileen Nowak
d11fd4210b Updated docs api links to be version-less 2019-07-25 15:17:23 +08:00
Aileen Nowak
496f873ac4
Updated links to docs (#10941)
no issue
2019-07-22 18:17:50 +08:00
Fabien O'Carroll
5da8da1879
Deleted unused pg.js module (#10928)
no-issue

This module was first created (AFAICT) in c09c20ad8d (diff-20a31f345ca2643b2602224678bb8d5b) and
has since undergone some filename renames and eslint refactors - we don't support
PostgreSQL and have no immediate plans to do so.
2019-07-22 12:11:19 +08:00
Naz Gargol
b89b57b6fb
Added global site SEO fields in Admin API (#10923)
refs #10921

- New SEO related fields will now be available as a part of Admin API /settings endpoint
- The ordering of fields is taken from post's schema
- Extended settings test suite with new SEO fields
- Adjusted settings model unit test
2019-07-18 16:24:34 +02:00
Naz Gargol
9dcc17a017
🐛 Fixed import for tag without slugs that belongs to a post (#10917)
closes #10785

- The behavior for tags will now be similar to posts' one described in the docs
- "The only strictly required field when importing posts is the title. Ghost will automatically generate slugs and set every other field to the default or empty."
- The breaking change was introduced with: 68d8154d4f (diff-e712df50c0dc7cf33746eeff0564003cR97) (assumed there's always slug in the imported object which is not true)
- Added originalIdMap to the importer base class to track id
substitution so it can be used when dealing with relational resource
updates
- Removed explicit use of 'this.stripProperties(['id']);' in
beforeImport of base class because we need to assign and remove the id
property in the same place to track this change
- Only calling 'this.stripProperties(['id']);' in
settings/trusted_domain imports as the method won't be called otherwise
- Expanded regression tests with new supported import case
2019-07-16 12:01:44 +02:00
renovate[bot]
db53ac0721 Update Test & linting packages (major) (#10858)
no issue 

- Updated Test & linting packages
- Updated use of hasOwnProperty
- Using Object.prototype.hasOwnProperty instead (ref. eslint.org/docs/rules/no-prototype-builtins)
- Removed already defined built-in global variable Intl
- Applied `--fix` with lint command on `core/test` folder
- The rules were broken because some of them were made stricter for `eslint: recommended` ruleset (ref. https://eslint.org/docs/user-guide/migrating-to-6.0.0#eslint-recommended-changes)
- Removed redundant global variable declarations to pass linting
2019-07-05 13:40:43 +02:00
Naz Gargol
be27db46eb
Extracted frontend code from redirects API controllers (#10798)
refs #10790

- The code was moved out of controllers to reduce the number of coupling points between the API controllers and "frontend" services
- A nice side effect of this move is a decreased amount of code that will need to be maintained and reusability between existing controllers
- Calling just a few methods from frontend services on API level makes it easier to abstract fronted away from API
2019-06-21 16:50:16 +02:00
Naz Gargol
df7e64fafa
Extracted frontend folder (#10780)
refs #10790

- Moved /core/apps into core/frontend
- Moved /core/server/helpers to /core/frontend/helpers along with /core/server/services/themes
- Changed helper location in overrides
- Moved /core/server/services/routing to /core/frontend/services
- Moved /core/server/services/url to /core/frontend/services
- Moved /core/server/data/meta to /core/frontend/meta
- Moved /core/server/services/rss to /core/frontend/services
- Moved /core/server/data/xml to /core/frontend/services
2019-06-19 11:30:28 +02:00
Naz Gargol
abda6e6338
Migrated to use url-utils from Ghost-SDK (#10787)
closes #10773

- The refactoring is a substitute for `urlService.utils` used previously throughout the codebase and now extracted into the separate module in Ghost-SDK
- Added url-utils stubbing utility for test suites
- Some tests had to be refactored to avoid double mocks (when url's are being reset inside of rested 'describe' groups)
2019-06-18 15:13:55 +02:00
Rish
a4f119cb7f Moved visibility utility from static model fn to Ghost-SDK
refs #10618

- Visibility methods don't belong on model, but are generic utils
- Used directly from ghost helper's visibility methods, cleans up core
- Removes direct model dependency of theme helper
- Updated `foreach_spec` to correct test data as per schema - visibility property cannot be empty
2019-06-09 13:12:04 +05:30
Nazar Gargol
f5544e7831 Migrated to use @tryghost/social-urls package
refs #10618

- /lib/social/urls was extracted into SDK to move more modules out of the core and reduce coupling of the theme layer
2019-06-06 17:10:13 +02:00
Naz Gargol
cb199b17b8
Decoupled asset hash calculation from package.json (#10774)
refs https://github.com/TryGhost/Ghost/issues/9414
refs c9b95b4bbd

- Removed package version from asset hash calculation
- Package version doesn't introduce any value when calculating a hash because Date.now() provides enough randomization on its own
2019-06-05 10:06:25 +02:00
Aileen Nowak
d086841f36
Updated link to Marketplace (#10754)
no issue

- Marketplace moved to https://ghost.org/marketplace
2019-05-22 16:55:28 +08:00
Aileen Nowak
79345f9030 🐛 Fixed Twittercard having wrong property when no feature image provided
closes #10706

- Fixed a bug where we weren't assigning `summary_large_image` as Twitter card property when no feature image is provided, but a dedicated Twitter image
- Updated test to reflect this case to be tested better
2019-05-06 12:51:43 +02:00
Fabien O'Carroll
a77190a312 Removed 2.16 member permission migration
no-issue

It turned out that due to the mismatch between `"members"` and
`"member"`, that not a single row was added to the database via this
migration. Clearing the file for simplicity.
2019-05-02 17:31:38 +02:00
Fabien O'Carroll
c70f5a0567 Copied member permission migration from 2.16->2.22
no-issue

The original migration read data from the fixtures, and ensured the
relations were made. Now the fixtures have been corrected, we can use
the same migration for 2.22
2019-05-02 17:31:38 +02:00
Fabien O'Carroll
0a67a25dbc Fixed member permission/role relations in fixtures
no-issue

The previous relations were setup to match against "members", which is
plural, but the object_type on the permission is "member". This was
causing the permissions to not be added as a relation to the role.
2019-05-02 17:31:38 +02:00
Nazar Gargol
27dae4022c 🐛 Fixed meta schema for 'page' context in dynamic routing
refs #10082

- When specifying an existing page as an allias for collection, e.g: `data: page.it-is-a-page` it was failing to generate metadata
2019-04-30 18:20:20 +02:00
Nazar Gargol
63337dd823 Changed context used in error to help
refs #10571

- The context should provide the context in which the error has happened instead current value was more suited to purposes of help
2019-04-23 09:31:24 +02:00
Nazar Gargol
809e610842 Added notification access persmission for users of Editor role
closes #9546

- Updated permissions  fixture for Editor role
- Added migration for permissions for Editor users
- Updated tests
2019-04-18 13:02:23 +02:00
Fabien O'Carroll
f9899cb8c4
Updated theme layer to use members-ssr (#10676)
* Removed support for cookies in members auth middleware

no-issue

The members middleware will no longer be supporting cookies, the cookie
will be handled by a new middleware specific for serverside rendering,
more informations can be found here:

https://paper.dropbox.com/doc/Members-Auth-II-4WP4vF6coMqDYbSMIajo5

* Removed members auth middleware from site app

no-issue

The site app no longer needs the members auth middleware as it doesn't
support cookies, and will be replaced by ssr specific middleware.

https://paper.dropbox.com/doc/Members-Auth-II-4WP4vF6coMqDYbSMIajo5

* Added comment for session_secret setting

no-issue

We are going to have multiple concepts of sessions, so adding a comment
here to be specific that this is for the Ghost Admin client

* Added theme_session_secret setting dynamic default

no-issue

Sessions for the theme layer will be signed, so we generate a random hex
string to use as a signing key

* Added getPublicConfig method

* Replaced export of httpHandler with POJO apiInstance

no-issue

This is mainly to reduce the public api, so it's easier to document.

* Renamed memberUserObject -> members

no-issue

Simplifies the interface, and is more inline with what we would want to export as an api library.

* Removed use of require options inside members

no-issue

This was too tight of a coupling between Ghost and Members

* Simplified apiInstance definition

no-issue

* Added getMember method to members api

* Added MembersSSR instance to members service

* Wired up routes for members ssr

* Updated members auth middleware to use getPublicConfig

* Removed publicKey static export from members service

* Used real session secret

no-issue

* Added DELETE /members/ssr handler

no-issue

This allows users to log out of the theme layer

* Fixed missing code property

no-issue

Ignition uses the statusCode property to forward status codes to call sites

* Removed superfluous error middleware

no-issue

Before we used generic JWT middleware which would reject, now the
middleware catches it's own error and doesn't error, thus this
middleware is unecessary.

* Removed console.logs

no-issue

* Updated token expirty to hardcoded 20 minutes

no-issue

This returns to our previous state of using short lived tokens, both for
security and simplicity.

* Removed hardcoded default member settings

no-issue

This is no longer needed, as defaults are in default-settings.json

* Removed stripe from default payment processor

no-issue

* Exported `getSiteUrl` method from url utils

no-issue

This keeps inline with newer naming conventions

* Updated how audience access control works

no-issue

Rather than being passed a function, members api now receives an object
which describes which origins have access to which audiences, and how
long those tokens should be allowed to work for. It also allows syntax
for default tokens where audience === origin requesting it. This can be
set to undefined or null to disable this functionality.

{
    "http://site.com": {
        "http://site.com": {
            tokenLength: '5m'
        },
        "http://othersite.com": {
            tokenLength: '1h'
        }
    },
    "*": {
        tokenLength: '30m'
    }
}

* Updated members service to use access control feature

no-issue

This also cleans up a lot of unecessary variable definitions, and some
other minor cleanups.

* Added status code to auth pages html response

no-issue

This was missing, probably default but better to be explicit

* Updated gateway to have membersApiUrl from config

no-issue

Previously we were parsing the url, this was not very safe as we can
have Ghost hosted on a subdomain, and this would have failed.

* Added issuer to public config for members

no-issue

This can be used to request SSR tokens in the client

* Fixed path for gateway bundle

no-issue

* Updated settings model tests

no-issue

* Revert "Removed stripe from default payment processor"

This reverts commit 1d88d9b6d73a10091070bcc1b7f5779d071c7845.

* Revert "Removed hardcoded default member settings"

This reverts commit 9d899048ba7d4b272b9ac65a95a52af66b30914a.

* Installed @tryghost/members-ssr

* Fixed tests for settings model
2019-04-16 16:50:25 +02:00
Fabien O'Carroll
b50cff8753 Updated import to never use {active,instsalled}_apps
no-issue

This is so that new imports will not contain old default, or invalid
values.
2019-04-16 11:19:17 +02:00
Fabien O'Carroll
7461869f13 Updated import to only show warning when apps exist
no-issue

This is so that someone with empty defaults for installed_apps and
active_apps, or someone without the defaults at all, will not see the
warning message when importing.
2019-04-16 11:19:17 +02:00
Fabien O'Carroll
f74d459dad Removed active and installed apps default settings
no-issue
2019-04-16 11:05:33 +02:00
Fabien O'Carroll
32a4798d76 Added skip of active/installed apps settings during import (#10681)
no issue
- the `{active,installed}_apps` settings related to a very old, minimally implemented, unreleased, and problematic approach to custom apps
- this is the first step towards full removal of the old "apps" concept

Credits: Kacper Szurek
2019-04-15 11:10:10 +01:00
Rishabh Garg
35c4da710a
🐛 Fixed sitemap generation with correct date and images value (#10668)
closes #10640

- Updated sitemap resources data to include certain fields
- Fixes sitemap date and images value
- Updated date handling for sitemap nodes
2019-04-09 12:43:07 +05:30
Katharina Irrgang
a2f7160499
Enhanced 2.18 detection (#10610)
no issue

- discovered another case
2019-03-13 23:40:50 +01:00
Fabien O'Carroll
ca16b197a2
Checked null & undefined in settings migration (#10609)
no-issue

This is to make sure we catch all falsy cases of values.
2019-03-13 23:35:23 +01:00
Fabien O'Carroll
406e7c04d2
Improved error messages/handling for 2.18 migrations (#10608)
no-issue

We do not want to throw errors for edgecases, so we lot a message
telling the user to double check their settings.
2019-03-13 22:03:54 +01:00
Fabien O'Carroll
a8debd8980
🐛 Fixed private blogging getting disabled after 2.17 migration (#10606)
no-issue

The 2.17 migration included a bug which set the `is_private`, `amp` and `force_i18n` setting values to `'false'` when they should have been `'true'`

We've reverted these changes by reading the most recent backup file, and setting the value to `'true'` if the backup has it set to `'true'` AND the current db has it set to false.

We've also amended the broken migration, so that it does not cause this issue for future installs
2019-03-13 21:35:19 +01:00
Fabien O'Carroll
56b6c633f6
Updated backup file names to include timestamp (#10607)
no-issue

Currently if you run two migrations on the same day, the backup is overwritten. This change adds the `HH-mm-ss` to the file name, meaning that you get a unique backup for each migration.
2019-03-13 21:06:05 +01:00
Katharina Irrgang
f64af762ef 🐛 Fixed "Cannot read property 'feature_image' of undefined" (#10602)
no issue

- refs https://forum.ghost.org/t/default-hbs-cannot-read-property-feature-image-of-undefined/6194
-
2019-03-13 18:13:52 +08:00
kirrg001
555dc2f180 🐛 Fixed {{meta_title}} output when using data: page.{slug} in routes
refs #10082

- meta_title output wrong meta title

Only solves meta_title outout for this use case:

```
routes:
  /:
    data: page.{slug}
    template: t
```
2019-03-12 12:34:16 +01:00
kirrg001
3b4edccf62 🐛 Fixed {{meta_description}} output when using data: page.{slug} in routes
refs #10599

- meta_description output wrong meta description

Only solves meta_description for this use case:

```
routes:
  /:
    data: page.{slug}
    template: t
```
2019-03-12 12:34:16 +01:00
kirrg001
fef0aa44d3 🐛 Fixed meta twitter_image output when using data: post.{slug} for routes
refs #10082

```
routes:
  /news/:
    data: post.news
```

The twitter_image was not available, because the context is [news, post] and the data is in `data.post`.
The context helper was incorrect. I think it is still not fully correct, but only focused on this use case.
The meta layer needs a full refactoring.
2019-03-12 12:13:47 +01:00
Naz Gargol
34fad7eaaf
Added Canonical URL support to posts&pages in Admin & Content API v2 (#10594)
refs #10593

- Added `canonical_url` field to post&pages resources in Admin & Content APIs
- Support for canonical URL on metadata layer (used in {{ghost_head}} helper)
- Made sure the new field is not accessible from API v0.1 
- Added handling same domain relative and absolute URLs
2019-03-12 17:51:29 +08:00
kirrg001
255b55cab5 Added proper importer regression tests
refs #10582
2019-03-11 20:07:48 +01:00
kirrg001
145a9456a2 Updated settings validations
refs #10582

- the db input formatter ensures we always forward "true" or "false" for boolean fields
2019-03-11 20:07:48 +01:00
kirrg001
f80a9429b9 Added migration to normalize booleans
refs #10582
2019-03-11 20:07:48 +01:00
Nazar Gargol
942324b60e 🐛 Fixed private blogging getting enabled after import
refs #10582

- Importer should do similar conversion introduced in 04c60b4ce1
2019-03-07 18:44:47 +08:00
Nazar Gargol
3124d23ccf 🐛 Fixed force_i18n import validation error
closes #10580

- The validation was failing because boolean values in settings can also be "0" and "1". 04c60b4ce1 explains the reason why these 2 new values are allowed
2019-03-07 16:42:11 +08:00
Naz Gargol
9810834f7b
Removed redundant errorDetail and context stringification in errors (#10572)
no issue

- Additional JSON.stringify call is redundant because it is already happening internally in Ignition (https://github.com/TryGhost/Ignition/blob/master/lib/logging/GhostLogger.js#L241)
- Left stringification in importer as is, because the use case there is also
putting errors into 'problems' array and seems like those values have to
be stringified
2019-03-07 12:58:44 +08:00
Rish
d62feca270 Added members permission migrations
no issue

- Added new fixtures for member permissions for Admin
2019-02-26 15:43:36 +07:00
Rish
20a898a986 Refactored members auth flow with dynamic settings
no issue

- Updated members auth flow UI
- Updated members settings and routing to be dynamic
2019-02-26 15:43:36 +07:00
Kevin Ansfield
8e9ade6357
Added migration to insert builtin Zapier integration (#10500)
no issue

A new Zapier app will be released that uses the v2 Admin API which means it will require an ApiKey that is linked to an Integration.

- adds a `type` column to the `integrations` table with the following types allowed:
  - `custom` (default) used by custom integrations added by users
  - `builtin` used by built-in integrations that have their own UI and won't show up in the "Custom Integrations" list
  - `internal` used by "internal" integrations such as the scheduler
- adds a `zapier` "builtin" integration to the fixtures
2019-02-15 14:48:21 +00:00
kirrg001
fde31b31ba Added permissions for actions including migration
refs #10431

- migration script to add permissions for actions
- restricted to owner & admin & integration role for now
- we will add permissions for other rules too, but we need add more granular restrictions
  - e.g. contributors can only read actions for posts which he created
2019-02-06 18:59:09 +01:00
kirrg001
c127b406fc Added actions table including migration
refs 10431

- add actions table
- add migration
2019-02-06 18:59:09 +01:00
notanengineercom
5bd509c873 🐛 Fixed removal of temp files left behind by importer
refs #10174

- Improved importer cleanUp method usage, so the cleanup is called in cases when there is an error during an import stage
- Simplified files to clean up tracking as removal of files is now partially handled in uploader middleware
2019-01-30 09:24:29 +00:00
Fabien O'Carroll
e9c3040786 💡Reverted "empty-string to NULL" db migration (#10430)
no-issue
- revert the migration because migrations that (potentially) touch every row should be kept to major releases where possible
  - the migration was safe to run and won't cause any problems for anyone who has already upgraded
- reversion keeps the migration file but changes the contents to a no-op so that `migrations` table state is the same for all users whether they migrated with the full migration or the no-op version
2019-01-29 14:41:04 +00:00
Fabien O'Carroll
0edacf3fc1
🐛 Added migration to update empty strings to null (#10428)
closes #10388

This migration finds all tables with nullable columns, it then loops through the tables and their nullable columns, updating each column to a null when its current value is an empty string.
2019-01-28 19:00:29 +01:00
Hannah Wolfe
909f9012ed
Updated docs links to best equivalents (#10386)
* Updated docs links to best equivalents
   - Our documentation has been overhauled, this updates the all the old links sprinkled through Ghost
* Update integrity hash
2019-01-17 06:57:37 +00:00
Fabien O'Carroll
cf1e4aa593
💡 Disabled v0.1 Public API by default (#10372)
refs #10364
2019-01-15 11:09:34 +01:00
Katharina Irrgang
30a0b1794a
Added calculated excerpt field to Content API v2 (#10326)
closes #10062

- return `post.excerpt` for Content API v2
- do not use `downsize`, because we might want to get rid of it if we drop v0.1 (downsize does not create good excerpts)
- simple substring of the plaintext field
2019-01-04 19:00:45 +01:00
Fabien O'Carroll
b6152684a8
Shortened content api key lengths (#10322)
* Lowered the min length for api keys to 26

* Updated ApiKey model to use shorter secrets for content keys
2019-01-04 19:39:54 +07:00
Hannah Wolfe
80f9765a35 Removed deprecated settings
refs #10318

- cheap and dirty way of removing settings we don't use
- rewritten the settingsFilter function as that was unnecessarily complex
- aslo fixed the require of default-settings in the importer
2019-01-04 12:17:32 +00:00
Hannah Wolfe
ef98c65040
Clean v2 Content API (#10329)
* Removed unused fields from v2 Content API

- We want to ship the v2 Content API as clean and lean as we can
- Many fields in the DB aren't actually used, we shouldn't return these values
- Other values aren't useful outside of Admin clients, and shouldn't be returned either

Fields removed:
- tags: created_at, updated_at, parent
- authors: locale, accessibility, tour
- posts: locale, author status, page
2019-01-04 11:21:21 +00:00
Naz Gargol
d3f3b3dc20
Added plugin based author and public tag models in API v2 (#10284)
refs #10124

- Author model returns only users that have published non-page posts
- Added a public controller for tags (should be extracted to separate Content API controller https://github.com/TryGhost/Ghost/issues/10106)
- Made resource configuration dynamic based on current theme engine
- This needs a follow-up PR with fixes to the problems described in the PR
2019-01-03 20:30:35 +01:00
kirrg001
75037bcb28 Added some deprecation notes for x_by fields
closes #10286

- we will come up with a new activity stream/actions concept soon
2019-01-03 17:14:52 +01:00
Greg Hart
261a8234e9 Added Slack username option. (#10285)
refs #9735.
2018-12-14 18:57:32 +07:00
Fabien O'Carroll
0bf5dd212a Added members related db changes and models
refs #10213
2018-12-11 11:53:55 +07:00
Dilan Guneratne
6bc658be9b 🐛Removed user reference warning from importer report if post is a draft (#10169)
refs #9584

- The Importer checks if a user reference is null. But if the post is a draft and published_by is null, we should ignore the user reference detection.
- This change will avoid showing an incorrect user reference warning in the importer report for draft posts.
2018-12-03 17:48:03 +01:00
Fabien O'Carroll
db34297a74
Added string size comment to schema.js (#10179)
no-issue
2018-11-19 17:29:31 +07:00
Rishabh Garg
75f243a3a2
Added migration for webhook permissions_roles fixtures (#10153)
closes #10152

- Added new migration for webhook permissions_roles relations
2018-11-13 14:35:37 +05:30
John O'Nolan
9f7295853f Changed casper.ghost.org to static.ghost.org references (#10109)
no issue

Assets moved from gh-pages to https://github.com/tryghost/static and hostname changed, redirects already in place. Can be tested on https://demo.ghost.io (image should all work fine, try visiting one directly to verify redirect works)
2018-11-07 15:50:38 +01:00
Katharina Irrgang
781f5523bf Removed logic for migration script (2.0/6-replace-fixture-posts.js) (#10081)
no issue

- we've reconsidered replacing the fixture posts
- this PR removes the logic from this script
2018-10-30 12:50:19 +07:00
Hannah Wolfe
275d82199f 🐛 Fixed cardWidth being lost on 2.0 imports (#10068)
no issue

- When importing Ghost 2.0 blogs into 2.0 blogs...
- The Koenig image card would lose it's cardWidth setting,
- because it'd be overridden by the imageStyle setting, which was null
- The importer previous _only_ kept the width if importing 1.0 blogs
2018-10-26 11:30:39 +02:00
Kevin Ansfield
71f0c08a34
Added edit webhook route to v2 Admin API (#10033)
no issue
- webhooks UI requires the ability to edit webhooks
- added `edit` permission for `webhook`
- added `edit` method to v2 webhook controller
- added `PUT /webhooks/:id` route to v2 Admin API routes
2018-10-19 18:35:55 +01:00
Nazar Gargol
49a729f77f Excluded sessions table from exporter (#9961)
refs #9865

- Made sessions table to be skipped during the export process
2018-10-17 18:29:24 +07:00
Rishabh Garg
c638863cb9 Updated webhooks schema and add relation (#10018)
* Extended webhooks schema/model and connected with integrations

refs #9942

- Updated webhooks schema with new columns - name, integration_id, secret, last_triggered_at, api_version
- Updated webhooks and integration model to map relationships
- Updated schema hash
- Updated test utils to exclude new webhooks columns for response comparison

* Added migration script for new webhooks columns

refs #9942

- Added migration script in 2.3 to add new columns to webhooks

* Updated schema hash

* Updated maxLength for api_version in schema

* Removed concurrency value from migration script

* Added defaults for webhooks model

* Added status field to webhooks for last trigger status

* Updated schema hash

* Fixed tests with status field

* Removed concurrency value in migration script

* Cleanup

* Updated schema with new fields

 - last_triggered_status, last_triggered_error
2018-10-17 18:17:13 +07:00
Katharina Irrgang
f8d244f060
🐛 Fixed migration from v1 to 2.2.1 (#9984)
closes #9983

- everything is described in the target issue
- this PR fixes both problems described in the issue
- TryGhost/Ghost-CLI#839 was raised to avoid this problem in the future
2018-10-11 09:04:47 +02:00
Katharina Irrgang
d699daeb35 🐛Fixed sitemap duplicates after routes.yaml upload (#9957)
closes #9956

- sitemap reset was missing
2018-10-08 10:29:21 +07:00
Fabien O'Carroll
4ecf7352b5 Added session_secret setting to schema (#9953)
closes #9952 
- required by the express-session middleware
2018-10-05 12:25:51 +01:00
Rish
91f8e03244 Added mobiledoc_revisions schema, migration script and updated tests 2018-10-02 19:15:23 +02:00
Rish
96488af21d Added command support for index to wrapper 2018-10-02 19:15:23 +02:00
Kevin Ansfield
1db3aefb9b
Set up schema and models for API Key authentication (#9904)
refs https://github.com/TryGhost/Ghost/issues/9865
- schema migrations
  - adds `integrations` and `api_keys` tables
  - inserts `integration` and `api_key` permissions and Administrator role relationships
  - inserts `Admin Integration` role and permissions
- adds `Integration` model
- adds `ApiKey` model
  - creates default secret if not given
  - hardcodes associated role based on key type
    - `admin` = `Admin API Client`
    - `content` = no role
- updates `Role` model to use `bookshelf-relations` for auto cleanup of permission relationships on destroy
2018-10-02 17:46:38 +01:00
Fabien O'Carroll
1d17f2aa91 Created session table migration (#9908)
refs #9865

- This table will be used for storing user sessions in
2018-09-27 09:51:46 +02:00
Daman Mulye
214d682ea3 Removed change frequency and priority fields from sitemap generator (#9771)
closes #9765

- Google says that change frequency and priority are not important when search engines crawl through pages. Therefore, these two properties are unimportant and removed from Ghost. Reference: https://www.seroundtable.com/google-priority-change-frequency-xml-sitemap-20273.html
- Credits to @damanm24
2018-09-24 19:16:31 +02:00
Sumedh Nimkarde
efd1587ee9 Switched to eslint-plugin-ghost (#9835)
refs #9834

- @TODO: the test env eslint needs to use the plugin, not part of this PR
2018-09-17 20:49:30 +02:00
Katharina Irrgang
91950cfea0
Optimised fixture replacement migration script (#9812)
no issue

- do only look for published old fixture posts
  - otherwise we detect draft old fixture post
  - and then we would replace them with published new fixture posts, which is not a very nice experience for the user
- ensure, if we have found all old fixture posts, replace all of them with the correct date
  - otherwise they are getting replaced and the date is "now"
- in general, this migration script is tricky and it tries to be smart, but there are so many cases we can run into
- to remember: the goal was to replace all old with new fixture posts (e.g. you just installed 1.25 and straight migrate to 2.0 - the old fixture posts should get replaced)
- added more protections to ensure we never delete custom posts using the same fixture post slugs
2018-08-22 13:35:58 +02:00
kirrg001
fd8502448a Updated fixture post content
no issue

- Organising your content

From: Here, the theme would assign the post publicly displayed tags of Blog - but it would also keep a private record of the post being tagged with #video.
To: Here, the theme would assign the post publicly displayed tags of News - but it would also keep a private record of the post being tagged with #video.
2018-08-16 12:13:24 +02:00
kirrg001
86e8b368b6 Fixed mobiledoc being null in migration script
no issue

- we stringify the mobiledoc field and this ends in 'null'
- the model layer won't detect 'null'
2018-08-16 12:13:24 +02:00
kirrg001
f024b3ebb5 Optimised error message in importer
no issue

- optimised error message if you are trying to import a Ghost 0.X export
- same applies to wordpress exports, i've raised in issue https://github.com/TryGhost/wp-ghost-exporter/issues/12
2018-08-16 12:13:24 +02:00
Katharina Irrgang
220d0456b4 🎨 Changed fixture data and default settings (#9778)
closes #9774, refs #9742

- added new fixture posts for Ghost 2.0
- added migration file to remove old fixture posts
  - only remove them if they are owned by the Ghost author and if they are tagged with getting-started
  - added new fixture posts if you had all (!) old fixture posts
  - ensure on rollback we remove the new fixture posts again
- updated default settings
2018-08-16 12:13:24 +02:00
kirrg001
f0b04d0ef1 Fixed 4-permalinks-setting migration script
no issue

- remove the special comment when migrating the routes.yaml file
2018-08-16 12:13:24 +02:00
Katharina Irrgang
1caa941d20 🔥 Removed demo post (#9769)
refs #9742

- removed demo post fixture for fresh blogs
- added migration to remove the demo post from the database
- on rollback: bring back demo post
2018-08-16 12:13:24 +02:00
kirrg001
8bb7088ba0 🔥 Removed permalink setting
refs #9742

- removed usage of single permalink setting
  - with dynamic routing this configuration does no longer makes sense
  - because you can configure your permalinks in the routes.yaml
  - furthermore you can have multiple collections with multiple permalinks
- removed @blog.permalinks
- do not export permalink setting
- do not import permalink setting
- permalink setting UI will be removed soon
- get rid of {globals.permalink} completely
- remove yaml in-built migration
- do not expose settings.permalinks via the private API
- do not allow to edit this setting
- keep phyiscal value in case a blog needs to rollback from v2 to v1
- sorted out when the routers should be created
  - ensure routes.yaml file doesn't get validated before Ghost is fully ready to start
2018-08-16 12:13:24 +02:00
kirrg001
52a9eebabf Changed comments and deprecation notes
refs #9742
2018-08-16 12:13:24 +02:00
Katharina Irrgang
c39df004dc Changes for Koenig and Ghost 2.0 (#9750)
refs #9742, refs #9724

- handle König Editor format for 2.0
- adapted importer to be able to import 1.0 and 2.0 exports
- added migration scripts
  - remove labs flag for Koenig
  - migrate all old editor posts to new editor format
- ensure we protect the code against mobiledoc or html field being null
- ensure we create a blank mobiledoc structure if mobiledoc field is null (model layer)
- ensure you can fully rollback 2.0 to 1.0
- keep mobiledoc/markdown version 1 logic to be able to rollback (deprecated code)
2018-08-16 12:13:24 +02:00
kirrg001
d518f23b48 Updated docs links
refs #9742

- rebase against master updated some docs links again
- go over code base again and double check that all docs links are correct
- 2.0 will become the latest version on our readme pages
2018-08-16 12:13:24 +02:00
kirrg001
23b4fd26c6 Moved knex-migrator execution into Ghost
refs #9742, refs https://github.com/TryGhost/Ghost-CLI/issues/759

- required a reordering of Ghost's bootstrap file, because:
  - we have to ensure that no database queries are executed within Ghost during the migrations
  - make 3 sections: check if db needs initialisation, bootstrap Ghost with minimal components (db/models, express apps, load settings+theme)
- create a new `migrator` utility, which tells you which state your db is in and offers an API to execute knex-migrator based on this state
- ensure we still detect an incompatible db: you connect your 2.0 blog with a 0.11 database
- enable maintenance mode if migrations are missing
- if the migration have failed, knex-migrator roll auto rollback
  - you can automatically switch to 1.0 again
- added socket communication for the CLI
2018-08-16 12:13:24 +02:00
kirrg001
90b56f925a Changed rollback messages
refs #9742

- use logging.warn
- make sure it's understandable that we are rolling back
2018-08-16 12:13:24 +02:00
kirrg001
ef355e281c ES6 migration: server/data/migrations/versions/2.0
refs #9742

- use ES6
2018-08-16 12:13:24 +02:00
kirrg001
0a556f95c7 Optimised logging for the new migration files
refs #9742
2018-08-16 12:13:24 +02:00
kirrg001
529bec8089 Renamed core/server/data/export to core/server/data/exporter
no issue

- i'vejust renamed the folder
- it makes more sense
  - data/importer
  - data/exporter
2018-08-16 12:13:24 +02:00
kirrg001
ed0dc191b2 Added database migration for amp -> comment_id rename
refs #9742

- rename column from amp -> comment_id
- iterate over all posts and ensure we use the resource id or the original amp value
- provide down hook to undo this change
2018-08-16 12:13:24 +02:00
kirrg001
050e026f8b Improved debug logs in importer
no issue
2018-08-16 12:13:24 +02:00
kirrg001
ad1e9ab038 🎨 Importer disallows LTS imports
refs #9742

- Ghost 2.0 won't support LTS imports
- only Ghost 1.0 imports
2018-08-16 12:13:24 +02:00
kirrg001
e99015e700 Refactored importer tests
no issue

- these tests were a mess
- we had many duplicated tests
- it was very hard to work with the exporter files
- i have tidied up the whole file
  - first section: all general importer tests based on the current Ghost version
  - second section: 1.0 tests
- everything is now JSON based (much easier to control)
2018-08-16 12:13:24 +02:00
kirrg001
7db0739296 Renamed amp post column to column_id
refs 9742

- when we've introduced Ghost 1.0, we have noticed that we broke Disqus comments
- Disqus comments use a unique identifier, which is the post id
- that means if you have exported your LTS content and imported it into 1.0, all resource identifiers are regenerated
- but the Disqus must use the original post resource id
- that's why we have imported the old post id and remembered it in the `amp` field 🤠
- that was the only field which was available and un-used
- now in Ghost 2.0, we would like to rename the `amp` field to `comment_id`
- string length: 50 (i thought it might be safer to not use 24 characters, because who knows if support other comment id's in the future)
2018-08-16 12:13:24 +02:00
kirrg001
3ed5087deb Released memory in importer as early as possible
no issue

- set bigger objects to null as soon as possible
- this will trigger the GC to free memory
2018-08-12 12:22:23 +02:00
kirrg001
a36191de5c Enabled useNullAsDefault for SQlite3
closes #9587

- when `useNullAsDefault` was invented in knex 0.10, it was a breaking change, that update/insert etc no longer set's null as default if a field is missing
- at this time we thought it only affects our test env, because the test generator doesn't generate all fields
- but turned out the importer is affected as well e.g. you import a post with missing fields
- the importer doesn't iterate over all fields and checks if the field is present or not
- as this only happens with SQlite3, we should enable `useNullAsDefault` by default
- you can still disable this option if you want, but not recommended
- the reason why knex added this breaking change was that some applications want "undefined" as value
- this is not the case in Ghost, so it's fine to make use of the default null behaviour
2018-08-11 13:52:03 +02:00
kirrg001
1ce504bb2d Fixed exporting extra tables for endpoint db/backup
refs #8719

- initial commit: 40c8eacd44
- we have forgotten that there is another endpoint which triggers an export (the backup endpoint)
- this endpoint needs to accept the new `include` query param as well (was missing)
2018-08-10 15:31:54 +02:00
kirrg001
6c18ceef25 Importer: Added protection for duplicate trusted domains
refs #8719
2018-08-08 19:00:23 +02:00
kirrg001
75cc60c20a Added option to import clients and trusted domains
refs #9742, refs #8719

- make it possible to import more tables (optional)
  - available tables: clients, trusted domains
- by default we won't import these tables, you have to tell Ghost using `include` (same syntax on export)
- we won't announce this ability for now (stays hidden)
2018-08-03 14:12:06 +02:00
kirrg001
40c8eacd44 Added option to export extra tables
refs #9742, refs #8719

- you can now use `include` to export extra tables e.g. `include=clients`
- admin client won't make use of this option yet, maybe later and optional
- we won't announce this new ability for now (stays hidden)
2018-08-03 14:11:45 +02:00
kirrg001
569018c2bc 🐛 Fixed 1.25 migration for html field being null
no issue

- it can happen that the `html` field is null
- the migration script will fail and rollback the changes automatically
2018-07-25 21:54:44 +02:00
Katharina Irrgang
6a9fe1e10c
🐛 Fixed 1.25 migration for mobiledoc field being null (#9752)
refs #9751

- the mobiledoc field can be null
- e.g. if you import a JSON with no markdown/mobiledoc or html field

The migration script for 1.25 had only the purpose to migrate existing Koenig Beta posts.
2018-07-25 11:49:04 +02:00
Katharina Irrgang
76b9a49eb8
🎨 Added Koenig Demo Post (#9747)
no issue

- add a new migration for 1.25 to insert the draft demo post for existing blogs
- ensure new blogs get the draft demo post as well
- tested on sqlite3 + mysql
- added handling if Ghost Author user doesn't exist anymore (fallback to owner user)
2018-07-24 14:37:17 +02:00
Kevin Ansfield
f57268daae
Koenig - Finalise Koenig HTML output and migrate existing content (#9741)
refs https://github.com/TryGhost/Ghost/issues/9742

We've identified some changes we need to make to the HTML output of the [new Koenig editor](
https://forum.ghost.org/t/koenig-editor-beta-release/1284/102) for future proofing and consistency across cards.

- the `<div class="kg-post">` wrapper around post content has been removed
- for image cards the `.kg-image-wide` and `.kg-image-full` classes have been changed to `.kg-width-wide` and `.kg-width-full` and applied to the `<figure>` element rather than the `<img>` element

Before:
```html
<div class="kg-post">
    <figure class="kg-image-card">
        <img class="kg-image kg-image-wide" src="...">
        <figcaption>example wide image</figcaption>
    </figure>
</div>
```

After:
```html
<figure class="kg-image-card kg-width-wide">
    <img class="kg-image" src="...">
    <figcaption>example wide image</figcaption>
</figure>
```
2018-07-23 12:23:02 +01:00
kirrg001
132df78940 Updated docs links
refs #9742

- Ghost 2.0 is coming
- all doc links in 1.0 must use concrete links e.g. docs.ghost.org/v1 or themes.ghost.org/v1.23.0/
- if we release Ghost 2.0, docs.ghost.org will show 2.0 docs
2018-07-20 23:49:16 +02:00
kirrg001
b335002269 🐛 Fixed Disqus comments on post preview pages
closes #9727

- this is a short term fix to proof that Disqus comments on preview pages no longer appear on other threads
- this is not a full solution to the problem
- the private API still returns /404/, which is right now inconsistent, but not critical in any way
- the url helper will now output the post preview url if you serve a draft/scheduled post
- this should register unique page urls at Disqus and ensure uniquness for threads
  - i still don't understand why the cross posting happens at all, because we also pass an unique identifier to Disqus (the post ID)
- it could be that comments, which are added on the preview page, won't appear on the published urls, because
  the published url !== preview url. I wasn't able to figure this out via testing or reading their docs
2018-07-11 00:20:55 +02:00
kirrg001
00cf043e15 Fixed missing defaults in model layer
no issue

- reported in the forum: https://forum.ghost.org/t/publishing-with-a-single-post-request-to-posts/1648
- the defaults are defined in two places
  1. on the schema level (defaults for the database)
  2. on the ORM (model layer)
- the defaults on the db layer are set correctly when inserting a new resource
- but if we don't apply all defaults on the model layer, it will happen that model events are emitted without the correct defaults
  - see comment in code base
  - it's caused by the fact that knex only returns the inserted resource id (probably caused by the fact knex has to support x databases)
- components/modules are listening on model events and expect:
  1. a complete set of attributes
  2. a complete set of defaults
  3. sanitized values e.g. bool, date
- this commit fixes:
  1. added missing defaults for user & post model
  2. sanitize booleans (0|1 => false|true)
  3. added tests to ensure this works as expected
  4. clarfies the usage of `defaults`

Regarding https://forum.ghost.org/t/publishing-with-a-single-post-request-to-posts/1648:
  - the post event was emitted with the following values {page: undefined, featured: undefined}
  - the urlservice receives this event and won't match the resource against collection filters correctly
  - NOTE: the post data in the db were correct
2018-06-26 16:35:23 +02:00
David Balderston
74a6a413ed Changed theme fixture post: point to forum instead of slack (#9703)
no issue

 - changed the default themes post to direct users to the Ghost forum
rather than slack
- updated the tests to reflect this change
2018-06-25 22:06:21 +02:00
kirrg001
a1b55509df Dynamic Routing Beta: collection name behaviour
refs #9601

Example:

```
collections:
  /podcast/:
    permalink: /{slug}/
```

- the name of the collection is remembered as `routerName` (in the case above: "podcast")
- the name of the collection is important for two things
  1. context value
  2. template name
- the context value is available for specific theme helpers e.g. is helper, body_class helper
- we auto-lookup the collection name in your theme e.g. podcast.hbs
- this logic does not apply to static routes
- if you define templates on your collection, they are stronger than the collection name
2018-06-21 20:59:43 +02:00
kirrg001
8a10826518 Protected error if routes.yaml file doesn't contain any collections
no issue

- if you define no collections, but a static route, it can happen that the target template to render
  makes use of the {{ghost_head}} helper
- the {{ghost_head}} helper tries to create the primary rss feed url
- at the moment: no collections, no primary rss feed url
- if we offer the option to define custom rss rules, this function might need an extension
2018-06-13 21:01:05 +02:00
Katharina Irrgang
7b0d5d465b 🐛 Fixed preview url and Zapier on subdirectory (#9683)
closes #9675

- with dynamic routing we have introduced a breaking change, which we have overseen
- Ghost does not return absolute urls, that's why the clients need to concat the blog url and the resource url
- with 1.24.0 Ghost returned resource urls including the subdirectory
- this caused trouble for e.g. zapier or the preview feature in the admin client
- revert breaking change and ensure we only expose resource urls without subdirectory
2018-06-12 16:36:58 +01:00
kirrg001
f943acea58 🐛 Fixed unable to publish a post after slack hook import
no issue

- reported here: https://forum.ghost.org/t/issues-with-unconfigured-slack-new-post-fails-with-can-not-find-property-url-of-undefined/1569
- the value of the slack settings was incorrect
- introduced in 1.24.1
- this adds a protection for invalid slack settings values
- this adds the correct code
- furthermore: in the next minor we will add a small migration script to correct any affected incorrect slack settings values
2018-06-11 11:08:08 +02:00
kirrg001
a66478576f 🎨 Importer no longer imports the slack hook
no issue

- from now on: you have to manually reconfigure your slack hook after importing your data
- we were running into trouble that Ghost had import slack hooks, because it can happen very fast
  that you are importing someone's slack hook
2018-06-07 10:16:54 +02:00
Katharina Irrgang
b392d1925a
Dynamic Routing Beta (#9596)
refs #9601

### Dynamic Routing

This is the beta version of dynamic routing. 

- we had a initial implementation of "channels" available in the codebase
- we have removed and moved this implementation 
- there is now a centralised place for dynamic routing - server/services/routing
- each routing component is represented by a router type e.g. collections, routes, static pages, taxonomies, rss, preview of posts
- keep as much as possible logic of routing helpers, middlewares and controllers
- ensure test coverage
- connect all the things together
  - yaml file + validation
  - routing + routers
  - url service
  - sitemaps
  - url access
- deeper implementation of yaml validations
  - e.g. hard require slashes
- ensure routing hierarchy/order
  - e.g. you enable the subscriber app
  - you have a custom static page, which lives under the same slug /subscribe
  - static pages are stronger than apps
  - e.g. the first collection owns the post it has filtered
  - a post cannot live in two collections
- ensure apps are still working and hook into the routers layer (or better said: and register in the routing service)
- put as much as possible comments to the code base for better understanding
- ensure a clean debug log
- ensure we can unmount routes
  - e.g. you have a collection permalink of /:slug/ represented by {globals.permalink}
  - and you change the permalink in the admin to dated permalink
  - the express route get's refreshed from /:slug/ to /:year/:month/:day/:slug/
  - unmount without server restart, yey
- ensure we are backwards compatible
  - e.g. render home.hbs for collection index if collection route is /
  - ensure you can access your configured permalink from the settings table with {globals.permalink}

### Render 503 if url service did not finish

- return 503 if the url service has not finished generating the resource urls

### Rewrite sitemaps

- we have rewritten the sitemaps "service", because the url generator does no longer happen on runtime
- we generate all urls on bootstrap
- the sitemaps service will consume created resource and router urls
- these urls will be shown on the xml pages
- we listen on url events
- we listen on router events
- we no longer have to fetch the resources, which is nice
  - the urlservice pre-fetches resources and emits their urls
- the urlservice is the only component who knows which urls are valid
- i made some ES6 adaptions
- we keep the caching logic -> only regenerate xml if there is a change
- updated tests
- checked test coverage (100%)

### Re-work usage of Url utility

- replace all usages of `urlService.utils.urlFor` by `urlService.getByResourceId`
  - only for resources e.g. post, author, tag
- this is important, because with dynamic routing we no longer create static urls based on the settings permalink on runtime
- adapt url utility
- adapt tests
2018-06-05 19:02:20 +02:00
Katharina Irrgang
b8c142747a
🐛Fixed uppercase tag associations on import (#9655)
no issue

- we sanitise any incoming slug on the model layer e.g uppercase -> lowercase
- and when importing e.g. an uppercase slug, the importer was trying to compare the uppercase slug with the sanitised slug
2018-05-28 23:58:06 +02:00
kirrg001
5f5f0021db 🔥 Drop Node v4 Support
no issue

- support ends today
- see https://github.com/nodejs/Release
- removed `use strict`
2018-05-01 14:06:18 +02:00
kirrg001
2667ad366c Show correct importer context objects
refs #9584

- object by reference 🎡
- we modify the object later on
- this has resulted in a wrong context output in the admin client
  - e.g. we've output an updated user reference
2018-04-18 00:14:46 +02:00
Aileen Nowak
23f59c341c Replaced routeKeywords in config with hard coded keywords (#9561)
no issue
- removed the `routeKeywords` property from the config and used hard coded keywords.
- removed `routeKeywords` from public configuration API endpoint, as it's no longer used in the Admin.
2018-04-17 10:36:05 +01:00
kirrg001
d209a4d013 🐛 Fixed importer bug: can't resolve authors relation
closes #9547

- you setup a blog with the following owner:
  - email: test@ghost.org
  - name: test
  - slug: test
- now you import a JSON db file, which holds the exact same owner
  - this owner won't be imported, because it's a duplicate
  - but the slug is different (!)
- the importer tries to find a matching existing user, but won't find anything
- the importer then send an empty authors array `post.authors=[]` into the model layer
- this is not allowed -> this would mean, you are actively trying to unset all authors
2018-04-10 01:10:06 +02:00
kirrg001
5dc2dddfc2 Fixed migration script 1.22 for invalid authors
no issue

- the previous commit will insert two post author relations if the author id of a post is invalid
- if a blog has an invalid author_id (which should be an edge case), we update the author id to the owner id
  - `posts_authors` are auto inserted in this case
2018-04-02 12:27:06 +02:00
Katharina Irrgang
7bbde460af
Speed up 1.22 migration script (#9541)
no issue

- insert `posts_authors` relation via knex
- massive speed improvement

e.g. 1500 posts

- before: ~1min
- after: ~10sec
2018-04-02 12:00:00 +02:00
Katharina Irrgang
40d0a745df Multiple authors (#9426)
no issue

This PR adds the server side logic for multiple authors. This adds the ability to add multiple authors per post. We keep and support single authors (maybe till the next major - this is still in discussion)

### key notes

- `authors` are not fetched by default, only if we need them
- the migration script iterates over all posts and figures out if an author_id is valid and exists (in master we can add invalid author_id's) and then adds the relation (falls back to owner if invalid)
- ~~i had to push a fork of bookshelf to npm because we currently can't bump bookshelf + the two bugs i discovered are anyway not yet merged (https://github.com/kirrg001/bookshelf/commits/master)~~ replaced by new bookshelf release
- the implementation of single & multiple authors lives in a single place (introduction of a new concept: model relation)
- if you destroy an author, we keep the behaviour for now -> remove all posts where the primary author id matches. furthermore, remove all relations in posts_authors (e.g. secondary author)
- we make re-use of the `excludeAttrs` concept which was invented in the contributors PR (to protect editing authors as author/contributor role) -> i've added a clear todo that we need a logic to make a diff of the target relation -> both for tags and authors
- `authors` helper available (same as `tags` helper)
- `primary_author` computed field available
- `primary_author` functionality available (same as `primary_tag` e.g. permalinks, prev/next helper etc)
2018-03-27 15:16:15 +01:00
CriticalRespawn
60386dbba5 🐛Fixed misspelled schema.org type for WebSite (#9526)
closes #9525

- updated the schema type to 'WebSite' as it is the correct spelling according to schema.org. See [here](http://schema.org/WebSite)
2018-03-23 10:50:28 +07:00
Mante Bridts
7ed822cc0c 🐛Fixed i18n for the {{meta_title}} helper (#9468)
closes #9466

- make 'page' in the meta title translatable through subexpression in the meta title helper
- e.g. `{{meta_title page=(t....)}}`
2018-03-19 09:48:07 +01:00
kirrg001
68d8154d4f Imported nested tags by foreign key
no issue

- replace logic for preparing nested tags
- if you have nested tags in your file, we won't update or update the target tag
- we simply would like to add the relationship to the database
- use same approach as base class
  - add `posts_tags` to target post model
  - update identifiers
  - insert relation by foreign key `tag_id`
- bump bookshelf-relations to 0.1.10
2018-02-20 09:56:45 +01:00
kirrg001
5a4dd6b792 Increased speed of importer
no issue

- change behaviour from updating user references after the actual import to update the user reference before the actual import
  - updating user references after the import is way less case intense
  - that was the initial decision for updating the references afterwards
  - but that does not play well with adding nested relations by identifier
- the refactoring is required for multiple authors
  - if we e.g. store invalid author id's, we won't be able to add a belongs-to-many relation for multiple authors
  - bookshelf-relations is generic and always tries to find a matching target before attching a model
  - invalid user references won't work anymore
- this change has a very good side affect
  - 17mb takes on master ~1,5seconds
    - on this branch it takes ~45seconds
  - also the memory usage is way lower and stabler
  - 40mb takes 1,6s (times out on master)
2018-02-20 09:56:45 +01:00
Katharina Irrgang
0aff9f33d9
Improved validation layer (#9427)
refs https://github.com/TryGhost/Ghost/issues/3658

- the `validateSchema` helper was a bit broken
  - if you add a user without email, you will receive a database error
  - but the validation error should catch that email is passed with null
- it was broken, because:
  - A: it called `toJSON` -> this can remove properties from the output (e.g. password)
  - B: we only validated fields, which were part of the JSON data (model.hasOwnProperty)
- we now differentiate between schema validation for update and insert
- fixed one broken import test
  - if you import a post without a status, it should not error
  - it falls back to the default value
- removed user model `onValidate`
  - the user model added a custom implementation of `onValidate`, because of a bug which we experienced (see https://github.com/TryGhost/Ghost/issues/3638)
  - with the refactoring this is no longer required - we only validate fields which have changed when updating resources
  - also, removed extra safe catch when logging in (no longer needed - unit tested)
- add lot's of unit tests to proof the code change
- always call the base class, except you have a good reason
2018-02-16 00:49:15 +01:00
kirrg001
ed4fde4f00 🐛 Fixed migrating from < 1.13 to 1.21
no issue

- discovered while testing
- the fixture utility needed a protection against non existent roles in the database
  - it tries to fetch the contributor role from the database, which does not exist yet
2018-02-07 12:31:21 +01:00
Austin Burdine
777247cbc7 Contributor Role (#9315)
closes #9314 

* added fixtures for contributor role
* update post api tests to prevent contributor publishing post
* update permissible function in role/user model
* fix additional author code in invites
* update contributor role migration for knex-migrator v3
* fix paths in contrib migration
* ensure contributors can't edit or delete published posts, fix routing tests [ci skip]
* update db fixtures hash
* strip tags from post if contributor
* cleanup post permissible function
* excludedAttrs to ignore tag updates for now (might be removed later)
* ensure contributors can't edit another's post
* migration script for 1.21
2018-02-07 10:46:22 +01:00
kirrg001
39ee95cc07 Make use of ES6 arrow functions in our data importer
no issue

- reduces the usage of `self`
2018-01-28 15:48:24 +01:00
kirrg001
2a10c83d92 Improved memory usage in importer
no issue

- returning and remembering the data, which was imported, is...
  - not required when using the API
  - not required when importing via script
  - required for tests
  - added an option to have control over it
- make more usage of local variables
  - the GC cannot tidy up variables, which are defined outside of a loop, but used in the loop
- try to keep less memory in process
  - reduce the number of properties we have to remember
2018-01-28 14:26:38 +01:00
kirrg001
5f9c3b92bd Improved Base importer constructor readability
no issue

- better differentiation between options and data
- better readability how to access required data from file
2018-01-28 13:35:21 +01:00
kirrg001
82bb3aaea1 Do not import unknown author id, fallback to owner id
no issue

- if you import a JSON file with a post, which has an unknown author,
  the target user was removed from the blog
- Ghost can handle this case and still succeeds with import
  - but we have stored an `author_id` in the database, which does not map to any user and won't map in the future
- this can trouble if we add support for multiple authors
  - currently, we only return the `author_id` to the client and the client can map with `author_id` with users fetched by the API
    - if it does not find a user, it just falls back to a different user
  - but multiple authors have to be included explicit (`include=authors`) and we will return a mapped (author_id => user) result
  - it won't be able to find the user, because we lookup the database
  - this would result in an error
- there is in general no reason to import (or store) an unknown/invalid `author_id` into the database
- on import, we show you a warning and you can choose a different author if you want
- solution: fallback to owner user and extend warning
  - it's not a behaviour change, you still can import unknown author id's and the import won't fail
  - but we ensure valid author id's
- updated test
- further more: returning `author={}` when requesting `include=author` could trouble with ember currently
  - it expects the author to be returned
2018-01-27 12:54:36 +01:00
David Wolfe
b69b9780a9 Backup endpoint improvements (#9365)
closes #9297

- backup endpoint returns JSON
- allows setting of export filename
- DRY up code
- the endpoint is not documented, no breaking change
2018-01-11 16:03:21 +01:00
kirrg001
ccb3b3de38 Fixed 1.20 migration script
no issue

- `models.Settings.destroy` only accepts one argument
2018-01-09 21:38:13 +01:00
Katharina Irrgang
5b77f052d9
Update Notification improvements (#9123)
closes #5071

- Remove hardcoded notification in admin controller
  - NOTE: update check notifications are no longer blocking the admin rendering
  - this is one of the most import changes
  - we remove the hardcoded release message
  - we also remove adding a notification manually in here, because this will work differently from now on
    -> you receive a notification (release or custom) in the update check module and this module adds the notification as is to our database

- Change default core settings keys
  - remove displayUpdateNotification
    -> this was used to store the release version number send from the UCS
    -> based on this value, Ghost creates a notification container with self defined values
    -> not needed anymore

- rename seenNotifications to notifications
  -> the new notifications key will hold both
     1. the notification from the USC
     2. the information about if a notification was seen or not
  - this key hold only one release notification
  - and n custom notifications

- Update Check Module: Request to the USC depends on the privacy configuration
  - useUpdateCheck: true -> does a checkin in the USC (exposes data)
  - useUpdateCheck: false -> does only a GET query to the USC (does not expose any data)
  - make the request handling dynamic, so it depends on the flag
  - add an extra logic to be able to define a custom USC endpoint (helpful for testing)
  - add an extra logic to be able to force the request to the service (helpful for testing)

- Update check module: re-work condition when a check should happen
  - only if the env is not correct
  - remove deprecated config.updateCheck
  - remove isPrivacyDisabled check (handled differently now, explained in last commit)

- Update check module: remove `showUpdateNotification` and readability
  - showUpdateNotification was used in the admin controller to fetch the latest release version number from the db
  - no need to check against semver in general, the USC takes care of that (no need to double check)
  - improve readability of `nextUpdateCheck` condition

- Update check module: refactor `updateCheckResponse`
  - remove db call to displayUpdateNotification, not used anymore
  - support receiving multiple custom notifications
  - support custom notification groups
  - the default group is `all` - this will always be consumed
  - groups can be extended via config e.g. `notificationGroups: ['migration']`

- Update check module: refactor createCustomNotification helper
  - get rid of taking over notification duplication handling (this is not the task of the update check module)
  - ensure we have good fallback values for non present attributes in a notification
  - get rid of semver check (happens in the USC) - could be reconsidered later if LTS is gone

- Refactor notification API
  - reason: get rid of in process notification store
    -> this was an object hold in process
    -> everything get's lost after restart
    -> not helpful anymore, because imagine the following case
      -> you get a notification
      -> you store it in process
      -> you mark this notification as seen
      -> you restart Ghost, you will receive the same notification on the next check again
      -> because we are no longer have a separate seen notifications object
  - use database settings key `notification` instead
  - refactor all api endpoints to support reading and storing into the `notifications` object
  - most important: notification deletion happens via a `seen` property (the notification get's physically deleted 3 month automatically)
    -> we have to remember a seen property, because otherwise you don't know which notification was already received/seen

- Add listener to remove seen notifications automatically after 3 month
  - i just decided for 3 month (we can decrease?)
  - at the end it doesn't really matter, as long as the windows is not tooooo short
  - listen on updates for the notifications settings
  - check if notification was seen and is older than 3 month
  - ignore release notification

- Updated our privacy document
- Updated docs.ghost.org for privacy config behaviour
- contains a migration script to remove old settings keys
2018-01-09 15:20:00 +01:00
kirrg001
eb0a11d53a Fixed import test: post duplication detection within a file to import
no issue

- with 29e143fa9a import queries no longer run in parallel
- this commit simply adds a small code snippet to reflect the importer behaviour

1) duplicate slugs *within* a file are getting ignored
2) existing posts in the database and posts to import with the same slug, result in duplicates

Further improvements regarding duplication detection will happen via #8717.
2018-01-03 13:34:15 +00:00
kirrg001
428008e63d Revert "🐛 Fixed importer duplicate detection for posts"
refs #8717

- we decided to not changing the current importer behaviour
- no slug duplication detection means, importing posts can result in duplicates
2018-01-03 13:34:15 +00:00
kirrg001
02bd71d0f5 🐛 Fixed importer duplicate detection for posts
closes #8717

- this is now required, because we run import queries sequentiell
- this code protects two cases:
  - you have duplicate slugs in the JSON file (the first get's inserted, the second get's ignored)
  - you have an existing slug in the database and you try to import the same slug, get's ignored
2018-01-03 00:07:41 +01:00
kirrg001
29e143fa9a 🐛 Fixed large imports
closes #9348

- do not run import with `Promise.all`
- with a large import file, we run an enormous amount of queries in parallel, which does not allow Node to cleanup memory
- tested with an 13mb import file
- requires bookshelf-relations 0.1.4
2018-01-02 23:21:45 +01:00
kirrg001
64626dedd1 Moved social utility to lib/social
refs #9178

- not 100% sure about this, but i think it makes right now the most sense
- we have already a url service and creating another lib/url is confusing at the moment
- i'll copy the last utility `makeAbsoluteUrls` to the url service for now
- see next commit for explanation (!)
2017-12-14 22:34:05 +01:00
kirrg001
b474fb0d16 Moved ghost-version to lib
refs #9178
2017-12-14 22:14:55 +01:00
kirrg001
fc5b4dd934 Moved image utils to lib/image
refs #9178

- i am not super happy about `const imageLib = require('../lib/image')`
- i don't really like the name `imageLib`
- but i had no better idea 😃
- if we use the same name in the whole project, it's very easy to rename the folder or the variable
2017-12-14 20:46:53 +01:00
kirrg001
bb06a8426d Moved tokens, url safe and safe string utility to lib/security
refs #9178

- we could now also move any crypto usages to lib/security, but no priority
- the main goal is to tidy up our utils folder
2017-12-14 13:38:00 +01:00
kirrg001
4fe7c9d7fc Moved validateRedirects helper to data/validation
refs #9178
2017-12-14 11:55:10 +01:00
Katharina Irrgang
192ebb1739
Moved labs, auth, permissions, settings, mail, themes to services (#9339)
refs #9178

- move tests as well
2017-12-14 03:01:23 +01:00
kirrg001
dac9ffc473 Removed leftover usages for globalUtils.sequence
refs #9178
2017-12-14 01:40:15 +01:00
kirrg001
660a407c16 Burn asset-hash utility
refs #9178

- Ghost uses the Node crypto lib always direct (require('crypto'))
- it doesn't make sense to outsource a single crypto statement (for the asset hash)
- we either have to write a crypto wrapper to avoid writing long crypto statements or we keep the direct usages for every case
- for now, wrapping the crypto calls into a lib/crypto has no priority
2017-12-14 01:33:01 +01:00
kirrg001
11a8245ca6 Added TODO to core/server/data/validation/index.js
no issue

- we are modifying the global validator dependency, which is bad
- see https://github.com/chriso/validator.js/issues/525#issuecomment-213149570
- we should create our own class and expose a singleton for the whole Ghost project
2017-12-14 01:15:12 +01:00
kirrg001
f83cbf6117 Moved pipeline/sequence to lib/promise
refs #9178

- continue with killing our global utils folder
- i haven't found any better naming for lib/promise
- so, require single files for now
- instead of doing `promiseLib = require('../lib/promise')`
- we can optimise the requires later
2017-12-13 22:20:02 +01:00
kirrg001
485c264c69 Use Promise support which was added in fs-extra 3.x
no issue

- now that we use bluebird globally, we can use the promise support from fs-extra
2017-12-13 20:57:11 +01:00
kirrg001
d83f474ff8 Always require fs-extra - consistency change
no issue
2017-12-13 20:57:11 +01:00
Katharina Irrgang
397400b4f8
Moved visibility utility to static model fn (#9327)
refs #9178

- this logic belongs to a static model helper
- the visibility property is a model property, the knowledge about the visibility values belongs to the model
- rename the functions, so they make more sense
2017-12-13 13:19:51 +01:00
kirrg001
2c2d1e93cc Fixed wrong require path in webhook-permissions
no issue

- the require path to the logging module was wrong
- because of ac2578b419
2017-12-12 15:23:07 +01:00
Katharina Irrgang
a98346e2ce Migration for webhook permissions (#9320)
refs https://github.com/TryGhost/Ghost/pull/9230
2017-12-12 14:12:34 +00:00
kirrg001
6f6c8f4521 Import lib/common only
refs #9178

- avoid importing 4 modules (logging, errors, events and i18n)
- simply require common in each file
2017-12-12 10:28:13 +01:00
kirrg001
ac2578b419 Moved errors,logging,i18n and events to lib/common
refs #9178
2017-12-12 10:28:13 +01:00
kirrg001
4265afe580 Moved utils/url.js to UrlService
refs #9178

- we have to take care that we don't end up in circular dependencies
  - e.g. API requires UrlService and UrlService needs to require the API (for requesting data)
- update the references
- we would like to get rid of the utils folder, this is/was the most complicated change
2017-12-11 20:05:33 +01:00
Katharina Irrgang
0bb81bb3c4
Bump knex-migrator to version 3.1.1 (#9199)
no issue

- adapt major changes of knex-migrator v3
- adapt migration scripts, simplify and add `down` (rollback) hook if possible
- clear Ghost cache after init hook (because of `knex-migrator migrate --init`)
- ensure db migrations work with the CLI
- updated troubleshooting guide (https://docs.ghost.org/v1/docs/troubleshooting#section-task-execute-is-not-a-function)

**For development only: Please ensure you run `npm i -g knex-migrator@latest` to update your global installation to v3. We always prefer the local installation, but v3 has modified and added binaries.**
2017-12-05 09:14:55 +01:00
Guillem Andreu
860b38a1a7 🐛 Fixed Sitemap when permalink contains Primary Tag (#9273)
closes #9272

- included tags in sitemap post-generator
2017-11-27 12:38:56 +01:00
Kevin Ansfield
bffb3dbd90
Webhooks support for subscriber events (#9230)
no issue

Support for http://resthooks.org style webhooks that can be used with Zapier triggers. This can currently be used in two ways:

a) adding a webhook record to the DB manually
b) using the API with password auth and POSTing to /webhooks/ (this is private API so not documented)

⚠️ only _https_ URLs are supported in the webhook `target_url` field 🚨

- add `webhooks` table to store event names and target urls
- add `POST` and `DELETE` endpoints for `/webhooks/`
- configure `subscribers.added` and `subscribers.deleted` events to trigger registered webhooks
2017-11-21 15:43:14 +00:00
Aileen Nowak
1bb9d4ff00 Set soft limits for blog title and description (#9250)
refs #8143

Add max length validations to settings:
- `blog.title`: 150 chars
- `blog.description`: 200 chars

The `validateSettings` fn in our validations checks for existing `validations` properties in our `default-settings.json` file, similar to other tables in our `schema.js`.
2017-11-16 13:58:22 +01:00
John O'Nolan
0d54326121 Give default Ghost user fixture a helpful bio (#9241)
no issue

Had a couple of people ask about how to delete welcome posts easily, so adding a bio to the default user to draw a little more attention to it
2017-11-14 10:23:38 +00:00
Aileen Nowak
a35c0c29e0 Added more database soft limits (#9225)
refs #8143

Sets soft limits for certain db fields:

- `posts`:
	- `title`: 255 chars (current hard limit: 2,000 chars)
	- `meta_title`: 300 chars (current hard limit: 2,000 chars)
	- `meta_description`: 500 chars (current hard limit: 2,000 chars)
- `users`:
	- `bio`: 200 chars (current hard limit: 65,535 chars)
	- `location`: 150 chars (current hard limit: 65,535 chars)
	- `meta_description`: 500 chars (current hard limit: 2,000 chars)
	- `meta_title`: 300 chars (current hard limit: 2,000 chars)
- `tags`:
	- `description`: 500 chars (current hard limit: 65,535 chars)
	- `meta_title`: 300 chars (current hard limit: 2,000 chars)
	- `meta_description`: 500 chars (current hard limit: 2,000 chars)

- same error message for isLength validator as for hard limits (more improvements are comming with https://github.com/TryGhost/Ghost/issues/6050)
- added more tests for importer
- added dynamic translation key handling for validators errors (isLength is only supported atm)
2017-11-09 15:22:20 +01:00
Hannah Wolfe
86c6cec433 Moved RSS module -> services & controllers
refs #5091, #9192, #9178

- Get the RSS module into a much better shape
- Controller -> /controllers/rss
- Remainder -> /services/rss
- Moved tests to match & updated requires
2017-11-08 08:09:44 +00:00
Hannah Wolfe
60fd98679f
Split RSS module into controller + other (#9224)
refs #5091, refs #9192

- This is similar to #9218, in that I'm revealing bits of code that are "controllers" in our codebase. As opposed to routes, services, renderers etc.
- This also reveals some code which is identical to the channels controller
- There is more to do here, but for now I've got the module split up, and the tests split and improved.
- Next I'll split RSS into controller + service, DRY up the controller code, etc
2017-11-07 20:00:03 +00:00
rambii
341948966d 🐛 Fixed custom_excerpt not being used in RSS feeds (#9219)
closes #9210

* use `custom_excerpt` before `meta_description` in rss
2017-11-07 10:16:14 +00:00
Hannah Wolfe
bcf5a1bc34
Switch to Eslint (#9197)
refs #9178

* Add eslint deps, remove old lint deps
* Add eslint config, remove old lint configs
* Config for server and tests are different
* Tweaked rules to suit us
* Fix linting in codebase - lots of indent changes.
* Fix a real broken test
2017-11-01 13:44:54 +00:00
kirrg001
9d4ed8c61c Replaced extract-zip-fork by extract-zip
no issue

- we had to fork the original repository at one point, because of slow maintenance
- maintenance is back now
- https://github.com/maxogden/extract-zip/pull/52 was merged and released
2017-10-31 12:11:59 +01:00
Hannah Wolfe
b5228f5c43 Rename blog -> site
- We're going to be moving towards this naming convention more
- Doing this now makes it easier to name other things
2017-10-26 18:09:56 +01:00
Aileen Nowak
c8cbbc4eb6 Improved password validation rules (#9171)
refs #9150 

- Moves the password length fn from `models/user` to `data/validation` where the other validator functions live.
- Added password validation rules. Password rules added:
   - Disallow obviously bad passwords: '1234567890', 'qwertyuiop', 'asdfghjkl;' and 'asdfghjklm' for example
   - Disallow passwords that contain the words 'password' or 'ghost'
   - Disallow passwords that match the user's email address
   - Disallow passwords that match the blog domain or blog title
   - Disallow passwords that include 50% or more of the same characters: 'aaaaaaaaaa', '1111111111' and 'ababababab' for example.
- Password validation returns an `Object` now, that includes an `isValid` and `message` property to differentiate between the two error messages (password too short or password insecure).
- Use a catch predicate in `api/authentication` on `passwordReset`, so the correct `ValidationError` will be thrown during the password reset flow rather then an `UnauthorizedError`.
- When in setup flow, the blog title is not available yet from `settingsCache`. We therefore supply it from the received form data in the user model `setup` method to have it accessible for the validation.
2017-10-26 11:01:24 +01:00
Hannah Wolfe
e659766f55 Move xmlrpc & slack to services (#9179)
refs #9178

- Introduce the /services/ folder
- Move xmlrpc there
- Move slack there
- In slack: remove a usage of the settings API that should use settingsCache
- In slack: Simplify the tests 
- Various tiny changes to move towards code consistency
2017-10-25 15:27:56 +01:00
Hannah Wolfe
ac3feb96d6 🐛 Prevents xmlrpc pings happening on import (#9165)
closes #9164

- check options.importing on xmlrpc
- also don't ping if private
- cleanup slack to work the same way
- update tests
- TODO: we need to prevent this event happening altogether
2017-10-23 18:30:33 +01:00
Aileen Nowak
d4b6390fd6 Improved importer logic for password in users (#9161)
refs #9150

- move data manipulation for importing users from `importers/data/users` to `model/user` for more consistency (see behaviour of post imports)
- changed importing logic in `onSaving` fn for user model:
   - when importing, we set the password to a random uid and don't validate, just hash it and lock the user
   - when importing with `importPersistUser` we check if the password is a bcrypt hash already and fall back to normal behaviour if not (set random password, lock user, and hash password)
   - don't run validations when importing
2017-10-19 10:43:52 +01:00
Katharina Irrgang
594b0c2d14 Custom post templates (#9073)
closes #9060

- Update `gscan` - it now extracts custom templates and exposes them to Ghost
- Add `custom_template` field to post schema w/ 1.13 migration
- Return `templates` array for the active theme in `/themes/` requests
- Users with Author/Editor roles can now request `/themes/`
- Front-end will render `custom_template` for posts if it exists, template priority is now:
  1. `post/page-{{slug}}.hbs`
  2. `{{custom_template}}.hbs`
  3. `post/page.hbs`
2017-10-10 13:36:35 +01:00
Hannah Wolfe
2941932ebe Removed hardcoded url structures from RSS (#9113)
refs #5091

- This removes hardcoded config to generate feed urls
- This means that RSS feeds work properly for custom channels
- Remaining assumption is that paginated feeds will end /pageNum/
- Added extra tests
- Don't pass through query params
2017-10-09 16:08:53 +02:00
Aileen Nowak
19167c1af2 🐛 Fixed author helper not returning the correct url (#9102)
closes #9101

With 506a0c3e9e we don't expose the `status` field for author context anymore, which is used to determine the correct URL for the `{{url}}` helper in https://github.com/TryGhost/Ghost/blob/master/core/server/data/schema/checks.js#L13

This fix uses the field `profile_image` instead and adds a missing test for author context to the `{{url}}` helper test.
2017-10-05 13:50:55 +02:00
Katharina Irrgang
7800ed3d8b Private RSS feed (#9088)
refs #9001

When a blog is in private mode there is now an unguessable URL that allows access to the RSS feed for internal use, commenting systems, etc.

- add public hash for private blogging
  - auto generate on bootstrap if missing
  - global hash, we can re-use in the future
- update private blogging middleware to detect the private RSS URL and rewrite it so that the normal rss route/code is used for display
- if a normal `/rss/` route is accessed with a private session return a 404
2017-10-05 11:07:32 +01:00
kirrg001
41c9db30b5 Debug: MySQL logging hook
no issue
2017-10-04 10:05:35 +01:00
Katharina Irrgang
d943fc7cc9 Allow Upload/Download of redirects.json (#9029)
refs #9028

- add two new endpoints for uploading/downloading the redirects (file based)
- reload/re-register redirects on runtime
- migration for 1.9 to add permissions for redirects download/upload
2017-09-21 16:01:03 +01:00
kirrg001
f478e4f9c8 🎨 Enabled Unsplash by default
refs https://github.com/TryGhost/Ghost/issues/8859

There are four cases:

- unsplash setting is empty (default), admin can enable the app by default (hardcoded isActive:true)
- unsplash settings are set, unsplash is disabled, admin detects that app was disabled on purpose
- unsplash setting is set, unsplash is enabled and has a key, app is enabled, old key get's ignored and overridden on the next save
- unsplash setting is set, unsplash is enabled and has no key, app is enabled
2017-09-20 11:44:47 +01:00
Aileen Nowak
0ce24b48bd Moved published_at creation to fixtures/utils (#8595)
no issue

- follow-up from #8573
- bove the hack that creates published_at values from the migration fn to our fixture util
2017-09-19 12:54:01 +02:00
Hannah Wolfe
4237446277 Misc cleanup & consistency amends (#9002)
no issue

- Consistent naming for postLookup
   - makes it easier to search and inspect the various usages
- Cleanup unneeded code
- Make res.render calls more consistent
- add some consistency to the calls to res.render
- Remove ancient reference to dataProvider
- Let's call it models everywhere now...
- Use consistent formatting across the API
- we're no longer using alignment in vars
- Misc other consistency changes in API
- always refer to local utils as apiUtils
- logical grouping of requires - dependencies, utils, "lib common" etc
- use xAPI to refer to API endpoints, e.g. mailAPI, settingsAPI for clarity
2017-09-12 17:31:14 +02:00
Aileen Nowak
eef7932e94 Refactor: fetch image dimensions from local file storage (#8900)
refs #8868

- Removed image-size in blog logo fn for meta data and made it synchronous
- Renamed `image-size-from-url.js` to `image-size.js` (incl. the test)
- Added second fn `getImageSizeFromFilePath` that reads from local file storage
- Added guard in `getImageSizeFromUrl` that checks if the image should be on local file storage and uses the new fn then instead
- Added a fn `fetchDimensionsFromBuffer` that takes the file buffer and returns an `imageObject` with dimensions.
- Added a new utils.js in `adapters/storage` for getting the file storage path
2017-09-05 14:13:22 +02:00
Austin Burdine
a9e668a949 🐛 Fixed only save imported post ids if amp field is empty (#8967)
closes #8963

- if an LTS export is imported into a 1.0 blog, then the 1.0 blog is
exported and re-imported into another 1.0 blog, any post ids from the
lts import were getting clobbered. This only saves the post id if the
amp field does not already exist
- add failing test that passes w/change
2017-09-04 09:48:56 +02:00
Kevin Ansfield
f7ce8d0e79 Fix lint error 2017-08-22 14:06:36 +01:00
Kevin Ansfield
2928f649a0 Fix "add-backup-client" migration
no issue
- use correct parameters for `addFixturesForModel` method
2017-08-22 13:51:44 +01:00
David Wolfe
c3fcb3105f Add ghost-backup client to trigger export (#8911)
no issue
- adds a ghost-backup client
- adds a client authenticated endpoint to export blog for ghost-backup client only
- allows some additional overrides during import
- allows for an import by file to override locking a user and double hashing the password
2017-08-22 11:15:40 +01:00