refs https://github.com/TryGhost/Ghost/issues/10477
The unsaved changes modal is displaying even when the post has been saved if images have been uploaded because the server is transforming absolute image urls to relative during input of the `mobiledoc` field but not transforming them back to absolute during output. The editor then thinks it's out of sync and shows the warning when trying to leave.
- `@tryghost/url-utils` has been updated with new methods for transforming URLs in mobiledoc content
- moves absolute->relative transformation from the API input serializers into the Post model
- transforms URLs in more fields for a more comprehensive transformation and fewer issues when re-configuring a site's domain
- previously there could be problems with internal links between posts not being transformed so you could change the url config to newdomain.com but links in post content would still be pointing to olddomain.com
- updates the API post output serializers to transform all modified fields
- drops the `?absolute_urls=true` param switch from the `canary` API post output serializer so that all URLs are output as absolute
- we're transforming more urls to relative when saving so this is necessary to ensure the unsaved changes modal is not triggered
- the query param isn't documented and will disappear in v3
no issue
- Moved default_content_visibility out of labs as we should be extra careful with what is exposed in the labs + it doesn't really belong there.
no issue
- Added 'labs' flag settings test
- Added test for default_content_visibility flag
- Default post's visibility takes into account values set in default_content_visibility setting
no-issue
This maps the stripe_info property to the MemberStripeInfo model, so
that we can update the member model, and correctly add/edit rows in the
members-stripe-info table.
no issue
- Updated test utilities to clearly identify both fields are not used in API responses
- Updated comment to remember clearning authors/author_id before releasing Ghost 4.0
refs #5151
refs #10737
- Removed all uses/references to post's "staticPages" filter
- It was only a feature specific to API v0.1 which doesn't have to take space in the codebase anymore
refs https://github.com/TryGhost/Ghost/issues/10922
- adds migrations to...
1. add `post.type` column
2. populate `post.type` column based on `post.page` value
3. drop `post.page` column
- updates all code paths to work with `post.type` in place of `post.page`
- adds `nql-map-key-values` transformer for mapping `page`->`type` in `filter` params when using the v2 API
- modifies importer to handle `post.page`->`post.type` transformation when importing older export files
NOTE: The post metadata table split is purely an internal optimization for v3 and doesn't require or expect any external actions including related API usage in v3
We keep running into issues adding new fields to the post table because there are too many fields making the post table "too wide". We have also hit MySQL limitations in how many bytes can be in a row (64kb) with post table.
In v3, we decided to split the 8 post fields (meta, twitter and og) used for meta data into a posts_meta table as these 8 fields are all "problem" `varchar` fields and make sense logically grouped together. The API layer is unaffected by the split as input/output serializers ensure the data flow works the same way as it was in v2. Only thing to note is json export in v3 will have slightly different structure with posts meta fields as separate.
- Creates new post_meta schema/table with 8 fields (2 meta_* , 3 twitter_* and 3 og_*)
- Update relations between post and post_meta table
- Update input/output serializers to keep existing API behavior
- Avoids new entry in post_meta table for post where all meta fields are null
- Keeps the current fields API param behavior
- Handles migration of existing posts to new table structure
- Updates importer/exporter to work seamlessly with table changes
no issue
- Partial rever of changes from https://github.com/TryGhost/Ghost/pull/11107/files#diff-792e00d413994563a1607b2be123da13L67-L68
- This code has effects how updated_at is behaving and broke "can\'t edit dates and authors of existing tag" test
- Based on bda76acba6 comment this removal should be moved into API layer but a more appropriate place then output serializer should be chosen (or original breaking test corrected)
no issue
- Removed redundant model code in users
- Removed v0.1 specific attribute removal on model layer for post
- Removed property deletions comments handled in serialization layer
- Removed unused token.added listener. users.last_seen is populated on middleware layer when a new access token is created
- Removed unneeded test for setting last_seen in users
no issue
- v0.1 is ☠️ so the access/refresh token based auth is no longer used
- removed all code related to the `accesstokens` and `refreshtokens` tables
- removed all `passport` related dependencies as it's no longer used
no issue
- v0.1 is ☠️ so there's no longer any use of client auth
- removes all code related to `clients` and `client_trusted_domains`
- noops the "add backup client" migration in 1.7 because the referenced fixture no longer exists causing migrations and consequently all regression tests to fail
no issue
- Drops `ghost_auth_access_token` and `ghost_auth_id` fields since not used anymore
- Adds migration for dropping these columns from users table
- Drops Auth strategy - `ghostStrategy` - since its not used anymore
no issue
- the column addition/removal can be too slow for large sites
- will be added back in 3.0
---
Revert "Fixed canary api for page/type column"
This reverts commit a5a7e7e919.
Revert "Updated frontend canary url config for page/type"
This reverts commit 19100ec5e6.
Revert "Updated canary api to handle type column correctly (#11006)"
This reverts commit c3e8ba0523.
Revert "Ensured `page` filter works in routes.yaml"
This reverts commit 9037c19e50.
Revert "Replaced usage of mongo util with nql-map-key-values"
This reverts commit 8c5f1d0ef0.
Revert "Added shared nql-map-key-values module"
This reverts commit ef4fd4b8ef.
Revert "Ensured page prop is present on content api response"
This reverts commit cfa0a0862b.
Revert "Fixed failing regression tests"
This reverts commit 9c2bb3811f.
Revert "Updated xmlrpc and slack service to use type column"
This reverts commit 44a02c7d36.
Revert "Updated v0.1 posts api to work with type column"
This reverts commit 2c81d7c914.
Revert "Removed updates to v0.1 specific code"
This reverts commit 08d83c1f53.
Revert "Added missing context from ValidationError"
This reverts commit cd45ab4f54.
Revert "Renamed page->type in the page&posts serializers"
This reverts commit df99e724e3.
Revert "Added mongo helper to input serializers"
This reverts commit fb8eadb4a8.
Revert "Passed mongoTransformer through to NQL"
This reverts commit 0ae3f0fdfc.
Revert "Permitted mongoTransformer option for read methods"
This reverts commit a89376bf26.
Revert "Updated the count plugin to reference the type column"
This reverts commit a52f15d3d3.
Revert "Updated hashes for db integrity check"
This reverts commit bb6b337be3.
Revert "Remove page column and remaining references"
This reverts commit 9d7190d692.
Revert "Added type column to data generator"
This reverts commit e59806cb45.
Revert "Removed references to page column in rss tests"
This reverts commit 04d0f855de.
Revert "Removed page column references in validation tests"
This reverts commit f0afbc5cc0.
Revert "Updated the post model to use the `type` column"
This reverts commit 1189bc823a.
Revert "Updated url service to use type column"
This reverts commit 61612ba8fd.
Revert "Updated the v2 api to deal with type column"
This reverts commit 57afb2de2b.
Revert "Added type property to post model defaults"
This reverts commit dc3345b1c5.
Revert "Added type property to the default post fixtures"
This reverts commit 82d8c38033.
Revert "Added type column to posts table"
This reverts commit 9b85fc6a69.
no-issue
* Removed redundant options from permittedOptions
The column option is already permitted at the Base model level.
* Remove defaultColumnsToFetch from Base model
* Removed defaultColumnsToFetch from Post model
refs #10922
This will allow us to pass through a customer transformer to replace
references to removed columns in the mongo query generated inside of NQL
* Simplified db controller permissions options
The existing objects were confusing because they did the same thing as
setting permissions to true, but gave the impressions that something
special was happening/required.
* Added DB Backup Integration Role
This will allow us to assign certain api_keys this role, in order to
automate db backups
* Allowed admin api_keys to have configurable roles
This will allow keys for the admin api to do customised things such as db export
* Added ghost-backup integration to fixtures
* Added migrations for DB Backup Integration and role
no issue
- Forced a filter on read and browse requests to the integrations endpoint to limit fetches to only "custom" and "builtin" integration types
- Expanded test coverage for "internal" integrations
no issue
- Updated Test & linting packages
- Updated use of hasOwnProperty
- Using Object.prototype.hasOwnProperty instead (ref. eslint.org/docs/rules/no-prototype-builtins)
- Removed already defined built-in global variable Intl
- Applied `--fix` with lint command on `core/test` folder
- The rules were broken because some of them were made stricter for `eslint: recommended` ruleset (ref. https://eslint.org/docs/user-guide/migrating-to-6.0.0#eslint-recommended-changes)
- Removed redundant global variable declarations to pass linting
closes#10789
* Updated keypair generation to use a memoised fn
This allows us to embed the members dynamic defaults in the object at
definition, and will allow us to only create the keypair when we need
it, in future.
* Added getDefaultValue fn to default setting obj
This will allow us to generate the default values when they're needed
rather than at boot time.
* Ensured dynamic defaults only generated when used
This replaces all the dynamic default values with functions to return
the values, and then calls (if required) that function inside the
getDefaultValue method of the setting object.
This reverts commit 64735693be.
- `rsa-keypair` is a binary dependency that was failing to install for a lot of users, reverting for now so we can look at alternative options for speeding up boot time
closes#10773
- The refactoring is a substitute for `urlService.utils` used previously throughout the codebase and now extracted into the separate module in Ghost-SDK
- Added url-utils stubbing utility for test suites
- Some tests had to be refactored to avoid double mocks (when url's are being reset inside of rested 'describe' groups)
refs #10618
- Visibility methods don't belong on model, but are generic utils
- Used directly from ghost helper's visibility methods, cleans up core
- Removes direct model dependency of theme helper
- Updated `foreach_spec` to correct test data as per schema - visibility property cannot be empty
* Removed support for cookies in members auth middleware
no-issue
The members middleware will no longer be supporting cookies, the cookie
will be handled by a new middleware specific for serverside rendering,
more informations can be found here:
https://paper.dropbox.com/doc/Members-Auth-II-4WP4vF6coMqDYbSMIajo5
* Removed members auth middleware from site app
no-issue
The site app no longer needs the members auth middleware as it doesn't
support cookies, and will be replaced by ssr specific middleware.
https://paper.dropbox.com/doc/Members-Auth-II-4WP4vF6coMqDYbSMIajo5
* Added comment for session_secret setting
no-issue
We are going to have multiple concepts of sessions, so adding a comment
here to be specific that this is for the Ghost Admin client
* Added theme_session_secret setting dynamic default
no-issue
Sessions for the theme layer will be signed, so we generate a random hex
string to use as a signing key
* Added getPublicConfig method
* Replaced export of httpHandler with POJO apiInstance
no-issue
This is mainly to reduce the public api, so it's easier to document.
* Renamed memberUserObject -> members
no-issue
Simplifies the interface, and is more inline with what we would want to export as an api library.
* Removed use of require options inside members
no-issue
This was too tight of a coupling between Ghost and Members
* Simplified apiInstance definition
no-issue
* Added getMember method to members api
* Added MembersSSR instance to members service
* Wired up routes for members ssr
* Updated members auth middleware to use getPublicConfig
* Removed publicKey static export from members service
* Used real session secret
no-issue
* Added DELETE /members/ssr handler
no-issue
This allows users to log out of the theme layer
* Fixed missing code property
no-issue
Ignition uses the statusCode property to forward status codes to call sites
* Removed superfluous error middleware
no-issue
Before we used generic JWT middleware which would reject, now the
middleware catches it's own error and doesn't error, thus this
middleware is unecessary.
* Removed console.logs
no-issue
* Updated token expirty to hardcoded 20 minutes
no-issue
This returns to our previous state of using short lived tokens, both for
security and simplicity.
* Removed hardcoded default member settings
no-issue
This is no longer needed, as defaults are in default-settings.json
* Removed stripe from default payment processor
no-issue
* Exported `getSiteUrl` method from url utils
no-issue
This keeps inline with newer naming conventions
* Updated how audience access control works
no-issue
Rather than being passed a function, members api now receives an object
which describes which origins have access to which audiences, and how
long those tokens should be allowed to work for. It also allows syntax
for default tokens where audience === origin requesting it. This can be
set to undefined or null to disable this functionality.
{
"http://site.com": {
"http://site.com": {
tokenLength: '5m'
},
"http://othersite.com": {
tokenLength: '1h'
}
},
"*": {
tokenLength: '30m'
}
}
* Updated members service to use access control feature
no-issue
This also cleans up a lot of unecessary variable definitions, and some
other minor cleanups.
* Added status code to auth pages html response
no-issue
This was missing, probably default but better to be explicit
* Updated gateway to have membersApiUrl from config
no-issue
Previously we were parsing the url, this was not very safe as we can
have Ghost hosted on a subdomain, and this would have failed.
* Added issuer to public config for members
no-issue
This can be used to request SSR tokens in the client
* Fixed path for gateway bundle
no-issue
* Updated settings model tests
no-issue
* Revert "Removed stripe from default payment processor"
This reverts commit 1d88d9b6d73a10091070bcc1b7f5779d071c7845.
* Revert "Removed hardcoded default member settings"
This reverts commit 9d899048ba7d4b272b9ac65a95a52af66b30914a.
* Installed @tryghost/members-ssr
* Fixed tests for settings model
- closes#10555
- Added a check to the user modal that the new owner is active
- Had to refactor Owner->Author unit test (also renamed it)
- Based on the first 2 lines, owner->editor change is attempted (hence the rename)
- Since both stubs return a 'modal' with owner role which means owner->owner change is actually attempted
- Now that there's a user status check, added the `status` property to the user receiving owernship
no-issue
As we're deprecating apps, filters are no longer used.
- Removed use of filters in helpers
- Removed use of filters from routing service
- Removed use of filters from rss service
- Removed use of filters in base model
refs #10593
- Added `canonical_url` field to post&pages resources in Admin & Content APIs
- Support for canonical URL on metadata layer (used in {{ghost_head}} helper)
- Made sure the new field is not accessible from API v0.1
- Added handling same domain relative and absolute URLs
refs #10582
- I don't think this is a good idea
- If a user passses "null", we should treat it as a string
- I am not aware of a use case why people have "null" in their database
- If people send "null" via the API, we should respect this and accept a string
no issue
- Additional JSON.stringify call is redundant because it is already happening internally in Ignition (https://github.com/TryGhost/Ignition/blob/master/lib/logging/GhostLogger.js#L241)
- Left stringification in importer as is, because the use case there is also
putting errors into 'problems' array and seems like those values have to
be stringified
no issue
- Reported here: https://forum.ghost.org/t/in-version-2-16-3-found-bug/6065/3
---
Admin Client sends false or true booleans for `is_private` key.
The settings table has two columns "key" and "value". And "value" is always type TEXT.
If you pass value=false, the db will transform this value into "0".
`settingsCache.get('is_private')` is then always true, even though the value is meant to be false.
We should add a migration in v3 and normalize all setting values to ensure consistent database values. Furthermore, we should improve the handling around settings values in general.
For now, we protect parsing values from DB, which we anyway need to transform the values into the correct data type, because we always save strings. This will protect values being stored as "false" or "1" or whatever.
closes#10512
- Removed field filtering in blog owner fetching because it didn't work before (fields weren't reduced) and now broke generated sql queries (ambiguous id field)
refs #10512
- Fixed ability to fetch specific fields when fetching tag resource by id
- Also only returning `url` field when specified in `fields` parameter
no issue
- if you html is NULL e.g. you create a draft post, we always set "html" to ""
- this get's marked as changed
- !this.get('html') was added for the 2.0 migration, because some posts had custom mobiledoc, but no html value
refs #9299
- `contextUser` returns a number and if the previous x_by is "1", then bookshelf marks it as changed ("1" !== 1)
- this is a left over from 0.x, because we still owner as id 1
- as soon as we fix 9299, we don't have to worry about this anymore, because we will fetch the owner id if we need it
no issue
- make use of filter instead of status=all or data.page
- nql was designed to filter data on database layer
- do not break v0.1
- we just got rid of the "status" query param, you should use the filter instead
- get rid of the ugly condition to remove page field if "fields" param was used
- allow filtering on model layer for "findOne"
- do not allow filtering for "findOne" on API layer for now
- the API controller defines what is allowed
- the model layer can allow more by default
- we can re-use the powerful filter logic without adding hacks
refs #10438, refs #10106
* Renamed existing pages ctrl
* Splitted posts & pages for Admin API v2
* Added pages JSON input schema for Admin API v2
* Removed single author for Content & Admin API v2
- single author is not documented
- single author usage is deprecated in v0.1
- single author usage is removed in API v2
* Splitted posts & postsPublic controller for v2
* Removed requirement to send `status=all` from Admin API v2
* Removed `status` option from pages Content API v2
* Removed `status` options from Users Admin API v2
no issue
A new Zapier app will be released that uses the v2 Admin API which means it will require an ApiKey that is linked to an Integration.
- adds a `type` column to the `integrations` table with the following types allowed:
- `custom` (default) used by custom integrations added by users
- `builtin` used by built-in integrations that have their own UI and won't show up in the "Custom Integrations" list
- `internal` used by "internal" integrations such as the scheduler
- adds a `zapier` "builtin" integration to the fixtures
refs #10438
- we now try to match by slug or id or email
- fallback to owner
- you cannot create a user via post endpoint
- Ghost uses the invite flow to add users
- get rid of `id` restriction on API level
refs #10438
- Additional check for present 'name' property before generating a
'slug'. Setting slug should not succeed and throw validation error in later
stages.
refs #10461
- do not break the existing webhooks by keeping both payload formats for subscribers events
- refactored webhooks service to run models through target API version
- added new events described in the target issue reference
- this refactoring & enhancement is undocumented, further breaking changes will happen because we are actively working on: https://github.com/TryGhost/Ghost/issues/10438
refs #10461
- the model layer (only post & user) fetches the model after update
- i assume it was added to ensure a response with all fields
- quick fixing it for now to ensure API layer can access ".wasChanged" to be able to decide if a request modified a resource or not
@NOTE: Bookshelf does not physically update a resource if nothing has changed.
no issue
- the model & api layer suffered from missing fields when creating resources
- usually there is only a handful of fields which are required to insert a resource
- the other fields are nullable and/or get defaults assigned
- the API only returned the configured default fields and the fields you have sent to the API
- this resulted in a response with missing fields
- if you have listend on "created" event, the same happend
- you received a model with missing fields
- we now set the undefined fields to null on purpose to ensure a full model for both cases
@NOTE:
There is no endpoint to serve webhooks (not for v0.1, not for v2).
Exposing the secret is required if an integration fetches it's api keys and it's webhooks.
The secret is currently un-used and not implemented.
no issue
- the event chain works like this:
- if a model registers an event, it get's triggered, because it's stronger than the base model
- but you have to call the base model to agree on a contract, because base model implements generic logic in event handlers
- this was inconsistently used
refs #10431
- the model layer triggers a couple of events on resource update
e.g. post to page -> post.deleted, post.added
- the resource_type must be always "post", because "page" is not an official model (Bookshelf won't be able to resolve the resource anymore)
- the action streams looks very confusion if you see deleted and added actions when toggling the post to a static page
- therefor the easiest approach for now is to only store actions for: added, edited, deleted
- and we will add the context information asap
- e.g. you will see that status was changed from "draft" to "published"
- we can also introduce extra published actions if we want
- relying on the internal event system right now makes things just more complicated and we want to keep it simple
refs #9248
- Bookshelf gives access to ".changed" before the update
- Discussion: https://github.com/bookshelf/bookshelf/issues/1943
- We also need to know what has changed after the update to be able to decide if we should trigger events
- Furthermore: Bookshelf cannot handle relation updates, it always marks relations as changed even though they did not change
- Bumped bookshelf-relations to be able to
- know if relations were updated
- ensure we unset relations on bookshelf's ".changed"
refs #10438
- integrations != users
- Ghost's assumption is: if you create a post, the primary author becomes the logged in user
- we have to require authors for integrations
- short fix and needs some more thoughts later
refs #10388
This updates the base model to retrieve column information, and explicitly set every property whose column is `nullable` and content is the empty string (`""`) to `null`
refs #9865
- Changed id passed for api_key to an object to be able to differenciate between admin and content api requests
- Added integration id to frame context
- Small refactoring of frame context initialization
refs #10286
- this is just a hotfix for v0.1
- we keep the x_by fields for now and deprecate them
- as soon as an integration updates/inserts a resource, we just store the owner ID
- we currently work on a new concept for v2
- v2 no longer exposes or uses x_by, see https://github.com/TryGhost/Ghost/pull/10294
- we need to iterate on this change, because we currently use the naming `context.api_key_id` in the auth/API layer
refs #9865
This updates all current permissible methods to use the new function
signature which includes the hasApiKeyPermissions parameter. It also
makes sure that the hasApiKeyPermissions argument is taken into account
whenever checking before returning a resolved promise.
To be continued. This is just a tiny part of the big picture. None of these changes are fully committed to stay as they are.
refs https://github.com/TryGhost/Ghost/issues/10124
- This PR introduced additional db calls in URL service due to the need for a model recalculation (we can't rely on the objects that come with events)
* Removed unused fields from v2 Content API
- We want to ship the v2 Content API as clean and lean as we can
- Many fields in the DB aren't actually used, we shouldn't return these values
- Other values aren't useful outside of Admin clients, and shouldn't be returned either
Fields removed:
- tags: created_at, updated_at, parent
- authors: locale, accessibility, tour
- posts: locale, author status, page
refs #10124
- Author model returns only users that have published non-page posts
- Added a public controller for tags (should be extracted to separate Content API controller https://github.com/TryGhost/Ghost/issues/10106)
- Made resource configuration dynamic based on current theme engine
- This needs a follow-up PR with fixes to the problems described in the PR
refs #10105
- `options.where` is an older deprecated logic
- before the filter language was invented, Ghost generates statements for knex
- if we want to replace GQL with NQL, we can't generate these statements
- they are not understood from NQL, because NQL uses mongo JSON
- go through usages and rewrite the statements
- invent `extraFilters` for now
- we need to keep the support for `status` or `staticPages` for now (API requirement)
- IMO both shortcuts in the extra filters should be removed in the future
This commit is required for https://github.com/TryGhost/Ghost/pull/10159!
* Added updateLastSeen method to user model
refs #10138
* Refactor codebase to use user.updateLastSeen
refs #10138
This is to ensure all updates go via the same method, meaning any
specific logic can be handled in one place, it also helps with grepping
the codebase to find where this occurs
* Created updateUserLastSeen middleware for v2 admin
refs #10138
This is intended to be used with the v2 admin api and _possibly_ the
content api, to give us an accruate report on thelast time a user access
a ghost instance.
* Wired updateUserLastSeen up to v2 Admin API
closes#10138
* Fixed broken test for v2 admin api
no-issue
This test was broken because it was incorrectly testing for a method to
be called exactly once - this was irrelevant to the functionality being
tested for.
* Updated user check method to set status to active
no-issue
* Debounced the updateUserLastSeen middlware an hour
no-issue
* Resolved some PR comments
refs #9865
* Added generic messaging for resource not found
* Ensured integration model uses transaction for writes
* Created POST /integrations endpoint
* Created GET /integrations/:id endpoint
* Created GET /integrations endpoint
* Created PUT /integrations/:id endpoint
* Created DELETE /integrations/:id endpoint
* Extended webhooks schema/model and connected with integrations
refs #9942
- Updated webhooks schema with new columns - name, integration_id, secret, last_triggered_at, api_version
- Updated webhooks and integration model to map relationships
- Updated schema hash
- Updated test utils to exclude new webhooks columns for response comparison
* Added migration script for new webhooks columns
refs #9942
- Added migration script in 2.3 to add new columns to webhooks
* Updated schema hash
* Updated maxLength for api_version in schema
* Removed concurrency value from migration script
* Added defaults for webhooks model
* Added status field to webhooks for last trigger status
* Updated schema hash
* Fixed tests with status field
* Removed concurrency value in migration script
* Cleanup
* Updated schema with new fields
- last_triggered_status, last_triggered_error
no-issue
With the new framework it is hard to handle 404 errors outside of the
serialization layer, this is because we cannot force destroy, edit or
findOne to error if the model is missing. This lets us do that.
* Stopped api key from assigning the 'Owner' role
refs #9865
We do not want api keys to be able to assign the Owner role to any other
key or user.
* Cleaned up Role model permissible method
no-issue
closes#9983
- everything is described in the target issue
- this PR fixes both problems described in the issue
- TryGhost/Ghost-CLI#839 was raised to avoid this problem in the future
closes#9927
- Added post model implementation to be able to store up to 10 versions of mobiledoc
- Bumped GQL to support filtering on the mobiledoc revision table
- Added tests ensuring new functionality works
refs #9865
Both the Post and the Author model implement the permissible method,
however the Post model does not abide by the signature of the
permissible method and add their own parameter "result" at the end.
This makes changes to the permissible method difficult as we have to
take into account multiple signatures.
This changes the Post model permissible method to the correct signature,
but still retains the current functionality. This will make it easier to
break up future permission related PR's so they can be reviwed easier
and faster!
refs #9866
- prep for v2
- you can better unit test the permissible function
- this avoids copying over the permission handling to v2 controller
- it was possible to move this logic into the model layer, because we now support `unsafeAttrs`
no issue
- now that we have a concept of `unsafeAttrs`, we can move the custom permissions to the invite model
Why doing now?
A) We won't copy this controller code to v2.
B) Makes it easier to unit test this behaviour
Moved URL attributes logic from the model into API layer
refs #9866
- Moved URL related attribute calculation for posts, users, and tags into API layer
- Added test coverage for url attributes in tags/authors/primary_tags/primary_authors
refs https://github.com/TryGhost/Ghost/issues/9865
- schema migrations
- adds `integrations` and `api_keys` tables
- inserts `integration` and `api_key` permissions and Administrator role relationships
- inserts `Admin Integration` role and permissions
- adds `Integration` model
- adds `ApiKey` model
- creates default secret if not given
- hardcodes associated role based on key type
- `admin` = `Admin API Client`
- `content` = no role
- updates `Role` model to use `bookshelf-relations` for auto cleanup of permission relationships on destroy
refs #9866
- Removed `toJSON` call in `findPage`
- Added JSON serialization on API layer
- Reason: model and api layer were coupled - all other model actions just returned the raw data and no specific format
- Corrected test suites to serialize fetched models to JSON
- Removed `absolute_urls` attribute from validOptions findPage methods as it's no longer needed in the data layer
- Changed 'include' test as this option is now tolerated and returns data
refs #9881
This is because when extending these methods, you need to know the
contents of the extraAllowedProperties to replicate it in the subclass,
breaking the principle of open/closed.
closes#9832
The API _should_ be returning absolute URLs for everything, 3rd party applications require absolute urls to read and display ghost data correctly. Currently they have to concat the blog url and the resource url, which is very uncomfortable.
Changing the public api like this would be considered a breaking change however so we've opted to put it behind a query parameter named `absolute_urls`.
closes#9822
- Fixed the post count issue for co authors
- Corrected and refactored tests related to users post count
- Consistency fix, because we return all posts where the author is primary or co author for the author page already
no issue
- we have to explicitly reset the previous `updated_at` field, because Bookshelf auto-updates this field on each update
- we have to extend the condition to avoid updating the `updated_by` field
- detect and respect `options.migrating`
refs #9742, refs #9724
- handle König Editor format for 2.0
- adapted importer to be able to import 1.0 and 2.0 exports
- added migration scripts
- remove labs flag for Koenig
- migrate all old editor posts to new editor format
- ensure we protect the code against mobiledoc or html field being null
- ensure we create a blank mobiledoc structure if mobiledoc field is null (model layer)
- ensure you can fully rollback 2.0 to 1.0
- keep mobiledoc/markdown version 1 logic to be able to rollback (deprecated code)
refs #9742
- rebase against master updated some docs links again
- go over code base again and double check that all docs links are correct
- 2.0 will become the latest version on our readme pages
refs 9742
- when we've introduced Ghost 1.0, we have noticed that we broke Disqus comments
- Disqus comments use a unique identifier, which is the post id
- that means if you have exported your LTS content and imported it into 1.0, all resource identifiers are regenerated
- but the Disqus must use the original post resource id
- that's why we have imported the old post id and remembered it in the `amp` field 🤠
- that was the only field which was available and un-used
- now in Ghost 2.0, we would like to rename the `amp` field to `comment_id`
- string length: 50 (i thought it might be safer to not use 24 characters, because who knows if support other comment id's in the future)
no issue
- discovered while testing
- SQLite3 has a restriction that you cannot query more than 999 SQL variables (e.g. fetch all tag relations for all posts)
- if you have more than 999 posts in your database, Ghost wasn't able to finish the bootstrap
- the url service tried to fetch all posts & relations, but could not complete, because SQlite3 has thrown
-> "too many SQL variables"
- i have added a recursive query strategy only for SQLite3
- use offset/limit
no issue
- add a new migration for 1.25 to insert the draft demo post for existing blogs
- ensure new blogs get the draft demo post as well
- tested on sqlite3 + mysql
- added handling if Ghost Author user doesn't exist anymore (fallback to owner user)
refs #9742
- Ghost 2.0 is coming
- all doc links in 1.0 must use concrete links e.g. docs.ghost.org/v1 or themes.ghost.org/v1.23.0/
- if we release Ghost 2.0, docs.ghost.org will show 2.0 docs
no issue
- reported in the forum: https://forum.ghost.org/t/publishing-with-a-single-post-request-to-posts/1648
- the defaults are defined in two places
1. on the schema level (defaults for the database)
2. on the ORM (model layer)
- the defaults on the db layer are set correctly when inserting a new resource
- but if we don't apply all defaults on the model layer, it will happen that model events are emitted without the correct defaults
- see comment in code base
- it's caused by the fact that knex only returns the inserted resource id (probably caused by the fact knex has to support x databases)
- components/modules are listening on model events and expect:
1. a complete set of attributes
2. a complete set of defaults
3. sanitized values e.g. bool, date
- this commit fixes:
1. added missing defaults for user & post model
2. sanitize booleans (0|1 => false|true)
3. added tests to ensure this works as expected
4. clarfies the usage of `defaults`
Regarding https://forum.ghost.org/t/publishing-with-a-single-post-request-to-posts/1648:
- the post event was emitted with the following values {page: undefined, featured: undefined}
- the urlservice receives this event and won't match the resource against collection filters correctly
- NOTE: the post data in the db were correct
refs #9601
- the home.hbs behaviour for the index collection (`/`) is hardcoded in Ghost
- we would like to migrate all existing routes.yaml files
- we only replace the file if the contents of the routes.yaml file equals the old routes.yaml format (with home.hbs as template)
- updated README of settings folder
- if we don't remove the home.hbs template from the default routes.yaml file, home.hbs will be rendered for any page of the index collection
- the backwards compatible behaviour was different
- only render home.hbs for page 1
- remember: the default routes.yaml file reflects how Ghost was working without dynamic routing
refs #9601
### Dynamic Routing
This is the beta version of dynamic routing.
- we had a initial implementation of "channels" available in the codebase
- we have removed and moved this implementation
- there is now a centralised place for dynamic routing - server/services/routing
- each routing component is represented by a router type e.g. collections, routes, static pages, taxonomies, rss, preview of posts
- keep as much as possible logic of routing helpers, middlewares and controllers
- ensure test coverage
- connect all the things together
- yaml file + validation
- routing + routers
- url service
- sitemaps
- url access
- deeper implementation of yaml validations
- e.g. hard require slashes
- ensure routing hierarchy/order
- e.g. you enable the subscriber app
- you have a custom static page, which lives under the same slug /subscribe
- static pages are stronger than apps
- e.g. the first collection owns the post it has filtered
- a post cannot live in two collections
- ensure apps are still working and hook into the routers layer (or better said: and register in the routing service)
- put as much as possible comments to the code base for better understanding
- ensure a clean debug log
- ensure we can unmount routes
- e.g. you have a collection permalink of /:slug/ represented by {globals.permalink}
- and you change the permalink in the admin to dated permalink
- the express route get's refreshed from /:slug/ to /:year/:month/:day/:slug/
- unmount without server restart, yey
- ensure we are backwards compatible
- e.g. render home.hbs for collection index if collection route is /
- ensure you can access your configured permalink from the settings table with {globals.permalink}
### Render 503 if url service did not finish
- return 503 if the url service has not finished generating the resource urls
### Rewrite sitemaps
- we have rewritten the sitemaps "service", because the url generator does no longer happen on runtime
- we generate all urls on bootstrap
- the sitemaps service will consume created resource and router urls
- these urls will be shown on the xml pages
- we listen on url events
- we listen on router events
- we no longer have to fetch the resources, which is nice
- the urlservice pre-fetches resources and emits their urls
- the urlservice is the only component who knows which urls are valid
- i made some ES6 adaptions
- we keep the caching logic -> only regenerate xml if there is a change
- updated tests
- checked test coverage (100%)
### Re-work usage of Url utility
- replace all usages of `urlService.utils.urlFor` by `urlService.getByResourceId`
- only for resources e.g. post, author, tag
- this is important, because with dynamic routing we no longer create static urls based on the settings permalink on runtime
- adapt url utility
- adapt tests
refs https://github.com/TryGhost/Ghost/issues/9505
- remove requirement for the `enableDeveloperExperiments` flag to be able to use Koenig
- it's now possible to enable as a standard Labs beta feature
refs https://github.com/TryGhost/Ghost/issues/9505
- updates mobiledoc converter's `render` method to accept a `version` argument
- `1` === Ghost 1.0's markdown-only renderer output
- `2` === Koenig's full mobiledoc renderer output
- switch between mobiledoc renderer versions in Post model's `onSaving` hook
- version 1 by default
- version 2 if Koenig is enabled (currently behind dev experiments config + labs flag)
- version 2 if the post's mobiledoc is not compatible with the markdown-only renderer
- "version 2" full-Koenig mobiledoc renderer output
- wraps content in a `.kg-post` div
- removes wrapper around markdown and html card output
- adds classes to image card output including selected image size/style
- standardises es6 usage across mobiledoc related files
refs https://github.com/TryGhost/Ghost/pull/9592
- we add bookshelf-relations step by step if we need it
- with https://github.com/TryGhost/Ghost/pull/9592 we have rewritten the test env to use Bookshelf
- this is important for our new url service
- because the service is listening on model updates and updates the urls based on the model events
- so with moving to Bookshelf, we need any easy way to add relations
- the test env inserts test fixtures
- it adds permissions and each permission get's roles attached
- `models.Permission.add({roles: [...]})
no issue
- replaced token creation by `lib.common.security`
- added unit tests for adding invites
- allow a different invite status for internal access
refs https://github.com/TryGhost/Team/issues/65
We are currently work on dynamic routing (aka channels).
An important piece of this feature is the url service, which always knows the url of a resource at any time.
Resources can belong to collections or taxonomies, which can be defined in a [routing yaml file](https://github.com/TryGhost/Ghost/issues/9528). We are currently shipping portions, which will at end form the full dynamic routing feature.
### Key Notes
- each routing type (collections, taxonomies, static pages) is registered in order - depending on the yaml routes file configuration
- static pages are an internal concept - they sit at the end of the subscriber queue
- we make use of a temporary [`Channels2`](https://github.com/TryGhost/Ghost/pull/9550/files#diff-9e7251409844521470c9829013cd1563) file, which simulates the current static routing in Ghost (this file will be modified, removed or whatever - this is one of the next steps)
- two way binding: you can ask for a resource url based on the resource id, you can ask for the resource based on the url
- in theory it's possible that multiple resources generate the same url: we don't handle this with collision (because this is error prone), we handle this with the order of serving content. if you ask the service for a resource, which lives behind e.g. /test/, you will get the resource which is served
- loose error handling -> log errors and handle instead of throw error and do nothing (we log the errors with a specific code, so we can react in case there is a bug)
- the url services fetches all resources on bootstrap. we only fetch and keep a reduced set of attributes (basically the main body of a resource)
- the bootstrap time will decrease a very little (depending on the amount of resources you have in your database)
- we still offer the option to disable url preloading (in your config `disableUrlPreload: true`) - this option will be removed as soon as the url service is connected. You can disable the service in case you encounter a problem
- **the url service is not yet connected, we will connect the service step by step. The first version should be released to pre-catch bugs. The next version will add 503 handling if the url service is not ready and it will consume urls for resources.**
----
- the url service generates urls based on resources (posts, pages, users, tags)
- the url service keeps track of resource changes
- the url service keeps track of resource removal/insert
- the architecture:
- each routing type is represented by a url generator
- a routing type is a collection, a taxonomiy or static pages
- a queue which ensures that urls are unique and can be owned by one url generator
- the hierarchy of registration defines that
- we query knex, because bookshelf is too slow
- removed old url service files + logic
- added temp channels alternative (Channels2) -> this file will look different soon, it's for now the temporary connector to the url service. Also the name of the file is not optimal, but that is not really important right now.
no issue
- required for model events
- otherwise you won't receive a full data set
- in worst case you have to re-fetch the post
- required for the url service
- the url service always needs relations (authors,tags) to be able to generate the url properly
@IMPORTANT
- no API change, we still return what you are asking for
- we first edit/add the resource
- then we fetch the data with the API options
- @TODO: this can be optimised and will improve performance
picking/selecting it from the insert/update response
- this is an internal change
closes#9547
- you setup a blog with the following owner:
- email: test@ghost.org
- name: test
- slug: test
- now you import a JSON db file, which holds the exact same owner
- this owner won't be imported, because it's a duplicate
- but the slug is different (!)
- the importer tries to find a matching existing user, but won't find anything
- the importer then send an empty authors array `post.authors=[]` into the model layer
- this is not allowed -> this would mean, you are actively trying to unset all authors
no issue
- if multiple queries run in a transaction, the model events are triggered before the txn finished
- if the txn rolls back, the events are anyway emitted
- the events are triggered too early
- solution:
- `emitChange` needs to detect that a transaction is happening
- it listens on a txn event to determine if events should be triggered
no issue
- add a big comment
- describe:
- how this works
- why this is in place
- what does currently not work
- and why it will work with channels
- @TODO:
- figure out how to disallow:
- `models.Post.findAll({columns: id})`
- `post.save(data)`
- this will trigger bookshelf events and model events
- url generation currently needs a set of attributes (e.g. slug, published_at)
- will be auto-fixed with channels, because you can call `urlService.getUrl(post.id)`
- but what doesn't get solved is our model events
- e.g. `emitChange` needs `post.get('page')` to determine if it's a page
no issue
- this is no longer needed for now
- it was anyway a little bit ugly to modify bookshelf's `changed` object
- if we want to change something about figuring out if a model has changed (including relations)
-> we probably need to override bookshelf
refs #9548
- we always receive date strings from the client in ISO format
- we ensure that we transform these strings into JS dates for comparison
- when the client sends relations, we need to ensure that relations are checked as well
- will only work for the post model for now, because this is the only model which uses `bookshelf-relations`
- added unit tests
- removed some model tests, which do the same
no issue
This PR adds the server side logic for multiple authors. This adds the ability to add multiple authors per post. We keep and support single authors (maybe till the next major - this is still in discussion)
### key notes
- `authors` are not fetched by default, only if we need them
- the migration script iterates over all posts and figures out if an author_id is valid and exists (in master we can add invalid author_id's) and then adds the relation (falls back to owner if invalid)
- ~~i had to push a fork of bookshelf to npm because we currently can't bump bookshelf + the two bugs i discovered are anyway not yet merged (https://github.com/kirrg001/bookshelf/commits/master)~~ replaced by new bookshelf release
- the implementation of single & multiple authors lives in a single place (introduction of a new concept: model relation)
- if you destroy an author, we keep the behaviour for now -> remove all posts where the primary author id matches. furthermore, remove all relations in posts_authors (e.g. secondary author)
- we make re-use of the `excludeAttrs` concept which was invented in the contributors PR (to protect editing authors as author/contributor role) -> i've added a clear todo that we need a logic to make a diff of the target relation -> both for tags and authors
- `authors` helper available (same as `tags` helper)
- `primary_author` computed field available
- `primary_author` functionality available (same as `primary_tag` e.g. permalinks, prev/next helper etc)
closes#9520
- it contains a dependency bump of the latest Bookshelf release
- Bookshelf introduced a bug in the last release
- see https://github.com/bookshelf/bookshelf/pull/1583
- see https://github.com/bookshelf/bookshelf/pull/1798
- this has caused trouble in Ghost
- the `updated_at` attribute was not automatically set anymore
---
The bookshelf added one breaking change: it's allow to pass custom `updated_at` and `created_at`.
We already have a protection for not being able to override the `created_at` date on update.
We had to add another protection to now allow to only change the `updated_at` property.
You can only change `updated_at` if you actually change something else e.g. the title of a post.
To be able to implement this check i discovered that Bookshelfs `model.changed` object has a tricky behaviour.
It remembers **all** attributes, which where changed, doesn't matter if they are valid or invalid model properties.
We had to add a line of code to avoid remembering none valid model attributes in this object.
e.g. you change `tag.parent` (no valid model attribute). The valid property is `tag.parent_id`.
If you pass `tag.parent` but the value has **not** changed (`tag.parent` === `tag.parent_id`), it will output you `tag.changed.parent`. But this is wrong.
Bookshelf detects `changed` attributes too early. Or if you think the other way around, Ghost detects valid attributes too late.
But the current earliest possible stage is the `onSaving` event, there is no earlier way to pick valid attributes (except of `.forge`, but we don't use this fn ATM).
Later: the API should transform `tag.parent` into `tag.parent_id`, but we are not using it ATM, so no need to pre-optimise.
The API already transforms `post.author` into `post.author_id`.
refs #9519
- `errors.models.posts.postNotFound` -> wrong
- `errors.models.post.postNotFound` -> correct
- the i18n lib just logs the error and falls back to a valid error key
- wrong i18n keys will never break Ghost
closes#9085
Fixes an issue, where the client sets image properties to `""` after deleting the image. This causes problems with the query filter (see https://github.com/TryGhost/GQL/issues/24), as they have to be `null`.
Added a check in the model layer saving method to set value to `null`, when the property is empty.
Affected models and properties:
- `posts`:
- `feature_image`
- `og_image`
- `twitter_image`
- `users`:
- `profile_image`
- `cover_image`
- `tags`:
- `feature_image`
no issue
- currently if you would like to edit a resource (e.g. post) and you pass an invalid model id, the following happens
- permission check calls `Post.permissible`
- the Post could not find the post, but ignored it and returned `userPermissions:true`
- then the model layer is queried again and figured out that the post does not exist
- A: there is no need to query the model twice
- B: we needed proper error handling for post and role model
no issue
- replace logic for preparing nested tags
- if you have nested tags in your file, we won't update or update the target tag
- we simply would like to add the relationship to the database
- use same approach as base class
- add `posts_tags` to target post model
- update identifiers
- insert relation by foreign key `tag_id`
- bump bookshelf-relations to 0.1.10
no issue
- change behaviour from updating user references after the actual import to update the user reference before the actual import
- updating user references after the import is way less case intense
- that was the initial decision for updating the references afterwards
- but that does not play well with adding nested relations by identifier
- the refactoring is required for multiple authors
- if we e.g. store invalid author id's, we won't be able to add a belongs-to-many relation for multiple authors
- bookshelf-relations is generic and always tries to find a matching target before attching a model
- invalid user references won't work anymore
- this change has a very good side affect
- 17mb takes on master ~1,5seconds
- on this branch it takes ~45seconds
- also the memory usage is way lower and stabler
- 40mb takes 1,6s (times out on master)
no issue
- otherwise we will have trouble in the future fetching relations by foreign key
- e.g. `tag_id: {id}`
- this won't work if we don't explicitly define the name of the keys
- bookshelf can't fulfil the request
- this does not change any behaviour, it just makes use of the ability to define the names of your foreign keys
refs https://github.com/TryGhost/Ghost/issues/3658
- the `validateSchema` helper was a bit broken
- if you add a user without email, you will receive a database error
- but the validation error should catch that email is passed with null
- it was broken, because:
- A: it called `toJSON` -> this can remove properties from the output (e.g. password)
- B: we only validated fields, which were part of the JSON data (model.hasOwnProperty)
- we now differentiate between schema validation for update and insert
- fixed one broken import test
- if you import a post without a status, it should not error
- it falls back to the default value
- removed user model `onValidate`
- the user model added a custom implementation of `onValidate`, because of a bug which we experienced (see https://github.com/TryGhost/Ghost/issues/3638)
- with the refactoring this is no longer required - we only validate fields which have changed when updating resources
- also, removed extra safe catch when logging in (no longer needed - unit tested)
- add lot's of unit tests to proof the code change
- always call the base class, except you have a good reason
no issue
- `isNew` does not work in Ghost, because Ghost does not use auto increment id's
- see https://github.com/bookshelf/bookshelf/issues/1265
- see https://github.com/bookshelf/bookshelf/blob/0.10.3/src/base/model.js#L211
- we only had one occurance, which was anyway redundant
- if you add a user, `hasChanged('password') is true
- if you edit a user and the password has changed, `hasChanged('password')` is true as well
NOTE #1:
1. We can't override `isNew` and throw an error, because bookshelf makes use of `isNew` as well, but it's a fallback if `options.method` is not set.
2. It's hard to re-implement `isNew` based on `options.method`, because then we need to ensure that this value is always set (requires a couple of changes)
NOTE #2:
If we need to differentiate if a model is new or edited, we should manually check for `options.method === insert`.
NOTE #3:
The unit tests are much faster compared to the model integration tests.
I did a comparision with the same test assertion:
- unit test takes 70ms
- integration test takes 190ms
no issue
- move password hashing and password comparison to lib/security/password
- added two unit test
- FYI: password hashing takes ~100ms
- we could probably mock password hashing in certain cases when unit testing
no issue
- this commit cleans up the usages of `include` and `withRelated`.
### API layer (`include`)
- as request parameter e.g. `?include=roles,tags`
- as theme API parameter e.g. `{{get .... include="author"}}`
- as internal API access e.g. `api.posts.browse({include: 'author,tags'})`
- the `include` notation is more readable than `withRelated`
- and it allows us to use a different easier format (comma separated list)
- the API utility transforms these more readable properties into model style (or into Ghost style)
### Model access (`withRelated`)
- e.g. `models.Post.findPage({withRelated: ['tags']})`
- driven by bookshelf
---
Commits explained.
* Reorder the usage of `convertOptions`
- 1. validation
- 2. options convertion
- 3. permissions
- the reason is simple, the permission layer access the model layer
- we have to prepare the options before talking to the model layer
- added `convertOptions` where it was missed (not required, but for consistency reasons)
* Use `withRelated` when accessing the model layer and use `include` when accessing the API layer
* Change `convertOptions` API utiliy
- API Usage
- ghost.api(..., {include: 'tags,authors'})
- `include` should only be used when calling the API (either via request or via manual usage)
- `include` is only for readability and easier format
- Ghost (Model Layer Usage)
- models.Post.findOne(..., {withRelated: ['tags', 'authors']})
- should only use `withRelated`
- model layer cannot read 'tags,authors`
- model layer has no idea what `include` means, speaks a different language
- `withRelated` is bookshelf
- internal usage
* include-count plugin: use `withRelated` instead of `include`
- imagine you outsource this plugin to git and publish it to npm
- `include` is an unknown option in bookshelf
* Updated `permittedOptions` in base model
- `include` is no longer a known option
* Remove all occurances of `include` in the model layer
* Extend `filterOptions` base function
- this function should be called as first action
- we clone the unfiltered options
- check if you are using `include` (this is a protection which could help us in the beginning)
- check for permitted and (later on default `withRelated`) options
- the usage is coming in next commit
* Ensure we call `filterOptions` as first action
- use `ghostBookshelf.Model.filterOptions` as first action
- consistent naming pattern for incoming options: `unfilteredOptions`
- re-added allowed options for `toJSON`
- one unsolved architecture problem:
- if you override a function e.g. `edit`
- then you should call `filterOptions` as first action
- the base implementation of e.g. `edit` will call it again
- future improvement
* Removed `findOne` from Invite model
- no longer needed, the base implementation is the same
refs #6103
- simplify `toJSON`
- `baseKey` was not used - have not find a single use case
- all the functionality of our `toJSON` is offered in bookshelf
- `omitPivot` does remove pivot elements from the JSON obj (bookshelf feature)
- `shallow` allows you to not return relations
- make use of `serialize`, see http://bookshelfjs.org/docs/src_base_model.js.html#line260
- fetching nested relations e.g. `users.roles` still works (unrelated to this refactoring)
> pick('shallow', 'baseKey', 'include', 'context')
We will re-add options validation in https://github.com/TryGhost/Ghost/pull/9427, but then with the official way: use `filterOptions`.
---
We return all fetched relations (pre-defined with `withRelated`) by default.
You can disable it with `shallow:true`.
