Commit Graph

258 Commits

Author SHA1 Message Date
kirrg001
a157c147af Added model.wasChanged helper
refs #10461

- helpful if you need to ask if the model was changed
- Bookshelf does not support this feature yet
- https://github.com/bookshelf/bookshelf/issues/1943
2019-02-07 22:05:07 +01:00
kirrg001
2eb1f2a507 Avoided inserting actions if nothing has changed
refs #10431

- I thought i have added this protection, but got removed via the simplification in cca3317a11
2019-02-07 20:49:35 +01:00
kirrg001
b25da62cca Ensured defaults when creating resources
no issue

- the model & api layer suffered from missing fields when creating resources
- usually there is only a handful of fields which are required to insert a resource
- the other fields are nullable and/or get defaults assigned
- the API only returned the configured default fields and the fields you have sent to the API
  - this resulted in a response with missing fields
- if you have listend on "created" event, the same happend
  - you received a model with missing fields
- we now set the undefined fields to null on purpose to ensure a full model for both cases

@NOTE:
There is no endpoint to serve webhooks (not for v0.1, not for v2).
Exposing the secret is required if an integration fetches it's api keys and it's webhooks.
The secret is currently un-used and not implemented.
2019-02-07 20:20:10 +01:00
kirrg001
1e90126760 Remembered changed fields on destroy
refs #9248
2019-02-07 11:35:00 +01:00
kirrg001
865366c7c8 Ensured consistency for event handlers in the model layer
no issue

- the event chain works like this:
  - if a model registers an event, it get's triggered, because it's stronger than the base model
- but you have to call the base model to agree on a contract, because base model implements generic logic in event handlers
- this was inconsistently used
2019-02-07 11:02:35 +01:00
kirrg001
cca3317a11 Simplfied action trigger in base model
refs #10431

- the model layer triggers a couple of events on resource update
  e.g. post to page -> post.deleted, post.added
- the resource_type must be always "post", because "page" is not an official model (Bookshelf won't be able to resolve the resource anymore)
- the action streams looks very confusion if you see deleted and added actions when toggling the post to a static page
- therefor the easiest approach for now is to only store actions for: added, edited, deleted
- and we will add the context information asap
  - e.g. you will see that status was changed from "draft" to "published"
- we can also introduce extra published actions if we want
- relying on the internal event system right now makes things just more complicated and we want to keep it simple
2019-02-06 22:21:06 +01:00
kirrg001
5f4cf42c14 Fixed tests
no issue
2019-02-06 21:36:09 +01:00
kirrg001
a8e0a173c2 Redefined context.integration
no issue

- we only have access to the ID at the moment
- we really don't want to end up in the same situation as with "context.user"
2019-02-06 21:36:09 +01:00
kirrg001
68bdcfc753 Fixed model._changed for creating resources
no issue

- discovered while testing
- the events are still triggered though for posts because .authors are added on creation
2019-02-06 21:36:09 +01:00
kirrg001
8bb2c7d3d5 Added model implementation for actions
refs #10431

- added actions model
- added logic in event emiiter to insert actions if supported
2019-02-06 21:36:09 +01:00
kirrg001
8b3336c84d Added condition to not trigger events if nothing has changed
refs #9248

- we no longer trigger events if the db was not changed
2019-02-03 13:02:31 +01:00
kirrg001
3289dc7619 Introduced model._changed
refs #9248

- Bookshelf gives access to ".changed" before the update
  - Discussion: https://github.com/bookshelf/bookshelf/issues/1943
- We also need to know what has changed after the update to be able to decide if we should trigger events
- Furthermore: Bookshelf cannot handle relation updates, it always marks relations as changed even though they did not change
- Bumped bookshelf-relations to be able to
  - know if relations were updated
  - ensure we unset relations on bookshelf's ".changed"
2019-02-03 13:02:26 +01:00
kirrg001
d44d93a19d Correct event debug logs in base model
no issue
2019-02-01 19:48:27 +01:00
kirrg001
01419ef8b3 Added base model debug log for events
no issue
2019-01-31 23:02:48 +01:00
kirrg001
d20bdfd9c1 Moved more allowed options for findPage to base model
no issue

- no need to define these options in each model
2019-01-29 23:29:06 +01:00
kirrg001
87bb3052a8 Moved allowed filter option to base model
no issue

- no need to define the filter option in each model
2019-01-29 22:57:11 +01:00
Fabien O'Carroll
95880dddeb
🐛 Updated base model to store null instead of empty string (#10425)
refs #10388 

This updates the base model to retrieve column information, and explicitly set every property whose column is `nullable` and content is the empty string (`""`) to `null`
2019-01-28 16:58:28 +01:00
Nazar Gargol
6318b65cab Changed context.api_key_id to an object containing key type information
refs #9865

- Changed id passed for api_key to an object to be able to differenciate between admin and content api requests
- Added integration id to frame context
- Small refactoring of frame context initialization
2019-01-24 17:22:58 +00:00
Katharina Irrgang
4acc375fb6
Bumped knex & bookshelf dependencies (#10404)
refs #9389, refs #9248

- https://github.com/bookshelf/bookshelf/releases/tag/0.14.0
- Bookshelf has fixed it's previous attr handling, see https://github.com/bookshelf/bookshelf/pull/1848
- SQlite3 double slashes was merged into knex and released 👻tgriesser/knex@c746dea
2019-01-21 21:53:11 +01:00
Katharina Irrgang
3df763009e
Added quick fix to handle integration upserts for x_by columns (#10295)
refs #10286

- this is just a hotfix for v0.1
- we keep the x_by fields for now and deprecate them
- as soon as an integration updates/inserts a resource, we just store the owner ID
- we currently work on a new concept for v2
- v2 no longer exposes or uses x_by, see https://github.com/TryGhost/Ghost/pull/10294
- we need to iterate on this change, because we currently use the naming `context.api_key_id` in the auth/API layer
2019-01-21 12:41:45 +01:00
Naz Gargol
df1ba8aee1
Added ability to notify and update url service about changes in related resources (#10336)
refs https://github.com/TryGhost/Ghost/issues/10124

- This PR introduced additional db calls in URL service due to the need for a model recalculation (we can't rely on the objects that come with events)
2019-01-08 09:48:53 +00:00
Naz Gargol
d3f3b3dc20
Added plugin based author and public tag models in API v2 (#10284)
refs #10124

- Author model returns only users that have published non-page posts
- Added a public controller for tags (should be extracted to separate Content API controller https://github.com/TryGhost/Ghost/issues/10106)
- Made resource configuration dynamic based on current theme engine
- This needs a follow-up PR with fixes to the problems described in the PR
2019-01-03 20:30:35 +01:00
kirrg001
75037bcb28 Added some deprecation notes for x_by fields
closes #10286

- we will come up with a new activity stream/actions concept soon
2019-01-03 17:14:52 +01:00
Dexter Leng
8c3d29edb2 🐛 Striped invisible unicode characters from slugs (#10252)
refs #9826

- Striped invisible characters from slug based on regexp pulled directly from XRegExp library, similarly to the approach taken in 9eef2616e4 (diff-50bf43eef863952813617ae9997538d0R4)
2018-12-13 13:06:59 +01:00
Katharina Irrgang
9d7c3bd726
🐛 Fixed all known filter limitations (#10159)
refs #10105, closes #10108, closes https://github.com/TryGhost/Ghost/issues/9950, refs https://github.com/TryGhost/Ghost/issues/9923, refs https://github.com/TryGhost/Ghost/issues/9916, refs https://github.com/TryGhost/Ghost/issues/9574, refs https://github.com/TryGhost/Ghost/issues/6345, refs https://github.com/TryGhost/Ghost/issues/6309, refs https://github.com/TryGhost/Ghost/issues/6158, refs https://github.com/TryGhost/GQL/issues/16

- removed GQL dependency
- replaced GQL with our brand new NQL implementation
- fixed all known filter limitations
- GQL suffered from some underlying filter bugs, which NQL tried to fix
- the bugs were mostly in how we query the database for relation filtering
- the underlying problem was caused by a too simple implementation of querying the relations
- mongo-knex has implemented a more robust and complex filtering mechanism for relations
- replaced logic in our bookshelf filter plugin
- we pass the custom, default and override filters from Ghost to NQL, which then are getting parsed and merged into a mongo JSON object. The mongo JSON is getting attached by mongo-knex.

NQL: https://github.com/NexesJS/NQL
mongo-knex: https://github.com/NexesJS/mongo-knex
2018-12-11 11:53:40 +01:00
Katharina Irrgang
e89a27f3ab
Replaced options.where GQL statements with filter notation (#10160)
refs #10105

- `options.where` is an older deprecated logic
- before the filter language was invented, Ghost generates statements for knex
- if we want to replace GQL with NQL, we can't generate these statements
- they are not understood from NQL, because NQL uses mongo JSON
- go through usages and rewrite the statements
- invent `extraFilters` for now
- we need to keep the support for `status` or `staticPages` for now (API requirement)
- IMO both shortcuts in the extra filters should be removed in the future

This commit is required for https://github.com/TryGhost/Ghost/pull/10159!
2018-11-15 15:53:24 +01:00
Katharina Irrgang
2e81852b22
Moved raw order for tag filtering to correct place (#10166)
refs #10105

- ordering !== filtering
2018-11-15 15:27:31 +01:00
Fabien O'Carroll
8046f4d437
🐛 Fix last seen for users (#10141)
* Added updateLastSeen method to user model

refs #10138

* Refactor codebase to use user.updateLastSeen

refs #10138

This is to ensure all updates go via the same method, meaning any
specific logic can be handled in one place, it also helps with grepping
the codebase to find where this occurs

* Created updateUserLastSeen middleware for v2 admin

refs #10138

This is intended to be used with the v2 admin api and _possibly_ the
content api, to give us an accruate report on thelast time a user access
a ghost instance.

* Wired updateUserLastSeen up to v2 Admin API

closes #10138

* Fixed broken test for v2 admin api

no-issue

This test was broken because it was incorrectly testing for a method to
be called exactly once - this was irrelevant to the functionality being
tested for.

* Updated user check method to set status to active

no-issue

* Debounced the updateUserLastSeen middlware an hour

no-issue

* Resolved some PR comments
2018-11-13 18:27:10 +07:00
kirrg001
976699336b Allowed columns options for findAll
no issue

- otherwise you can't filter the query
2018-10-12 21:13:20 +02:00
Fabien O'Carroll
48ebbf94d3
Permitted require option for edit, destroy, findOne (#9948)
no-issue

With the new framework it is hard to handle 404 errors outside of the
serialization layer, this is because we cannot force destroy, edit or
findOne to error if the model is missing. This lets us do that.
2018-10-12 18:00:21 +07:00
Katharina Irrgang
a7b0029471 Added mobiledoc revisions functionality
closes #9927

- Added post model implementation to be able to store up to 10 versions of mobiledoc
- Bumped GQL to support filtering on the mobiledoc revision table
- Added tests ensuring new functionality works
2018-10-09 15:31:09 +02:00
Fabien O'Carroll
fd84565218 Added check for orderDefaultOptions in base model (#9941)
no-issue

This is because calling findPage without an explicit order and not
defining an orderDefaultOptions throws.
2018-10-03 16:27:33 +02:00
Nazar Gargol
82b7aea641
Refactor URL generation from models (#9917)
Moved URL attributes logic from the model into API layer 

refs #9866

- Moved URL related attribute calculation for posts, users, and tags into API layer
- Added test coverage for url attributes in tags/authors/primary_tags/primary_authors
2018-10-03 15:44:30 +02:00
Fabien O'Carroll
15da580016 Added processOptions check to base model findPage
no-issue

When calling findPage on a model with no processOptions defined this
throws an error.
2018-10-03 10:16:10 +02:00
Fabien O'Carroll
ecf47f3b7b Added some missing tests to base model (#9931)
no-issue
2018-10-02 16:43:19 +02:00
Fabien O'Carroll
af12f21db7 Updated base model destroy fn to be more generic (#9928)
refs #9914

This allows us to destroy models on properties other than the id.
2018-10-01 14:44:52 +02:00
Nazar Gargol
4c5bff0f49 Removed toJSON serialization in findPage method (#9899)
refs #9866

- Removed `toJSON` call in `findPage`
- Added JSON serialization on API layer
- Reason: model and api layer were coupled - all other model actions just returned the raw data and no specific format
- Corrected test suites to serialize fetched models to JSON
- Removed `absolute_urls` attribute from validOptions findPage methods as it's no longer needed in the data layer
- Changed 'include' test as this option is now tolerated and returns data
2018-09-26 14:11:22 +02:00
Fabien O'Carroll
b326cfaab7 Updated base model to remove extraAllowedProperties
refs #9881

This is because when extending these methods, you need to know the
contents of the extraAllowedProperties to replicate it in the subclass,
breaking the principle of open/closed.
2018-09-21 12:15:29 +02:00
Sumedh Nimkarde
efd1587ee9 Switched to eslint-plugin-ghost (#9835)
refs #9834

- @TODO: the test env eslint needs to use the plugin, not part of this PR
2018-09-17 20:49:30 +02:00
Fabien O'Carroll
c9b8ddde4b 🎨Added absolute_url flag to public api (#9833)
closes #9832

The API _should_ be returning absolute URLs for everything, 3rd party applications require absolute urls to read and display ghost data correctly. Currently they have to concat the blog url and the resource url, which is very uncomfortable.

Changing the public api like this would be considered a breaking change however so we've opted to put it behind a query parameter named `absolute_urls`.
2018-08-31 11:02:39 +01:00
Katharina Irrgang
4b2ebee67b
🐛 Added logic to avoid updating the updated_at and updated_by field when migrating (#9814)
no issue

- we have to explicitly reset the previous `updated_at` field, because Bookshelf auto-updates this field on each update
- we have to extend the condition to avoid updating the `updated_by` field
- detect and respect `options.migrating`
2018-08-22 13:57:12 +02:00
kirrg001
3ed5087deb Released memory in importer as early as possible
no issue

- set bigger objects to null as soon as possible
- this will trigger the GC to free memory
2018-08-12 12:22:23 +02:00
kirrg001
08f54d2f57 Added protection against "too many SQL variables" for SQLite3
no issue

- discovered while testing
- SQLite3 has a restriction that you cannot query more than 999 SQL variables (e.g. fetch all tag relations for all posts)
- if you have more than 999 posts in your database, Ghost wasn't able to finish the bootstrap
  - the url service tried to fetch all posts & relations, but could not complete, because SQlite3 has thrown
    -> "too many SQL variables"
- i have added a recursive query strategy only for SQLite3
  - use offset/limit
2018-08-08 19:17:01 +02:00
Katharina Irrgang
76b9a49eb8
🎨 Added Koenig Demo Post (#9747)
no issue

- add a new migration for 1.25 to insert the draft demo post for existing blogs
- ensure new blogs get the draft demo post as well
- tested on sqlite3 + mysql
- added handling if Ghost Author user doesn't exist anymore (fallback to owner user)
2018-07-24 14:37:17 +02:00
kirrg001
61db6defde Added debug log to Base model: emitChange
no issue

- helpful for debugging model events
- DEBUG=ghost:models:base node index.js
2018-06-26 15:26:49 +02:00
Katharina Irrgang
b392d1925a
Dynamic Routing Beta (#9596)
refs #9601

### Dynamic Routing

This is the beta version of dynamic routing. 

- we had a initial implementation of "channels" available in the codebase
- we have removed and moved this implementation 
- there is now a centralised place for dynamic routing - server/services/routing
- each routing component is represented by a router type e.g. collections, routes, static pages, taxonomies, rss, preview of posts
- keep as much as possible logic of routing helpers, middlewares and controllers
- ensure test coverage
- connect all the things together
  - yaml file + validation
  - routing + routers
  - url service
  - sitemaps
  - url access
- deeper implementation of yaml validations
  - e.g. hard require slashes
- ensure routing hierarchy/order
  - e.g. you enable the subscriber app
  - you have a custom static page, which lives under the same slug /subscribe
  - static pages are stronger than apps
  - e.g. the first collection owns the post it has filtered
  - a post cannot live in two collections
- ensure apps are still working and hook into the routers layer (or better said: and register in the routing service)
- put as much as possible comments to the code base for better understanding
- ensure a clean debug log
- ensure we can unmount routes
  - e.g. you have a collection permalink of /:slug/ represented by {globals.permalink}
  - and you change the permalink in the admin to dated permalink
  - the express route get's refreshed from /:slug/ to /:year/:month/:day/:slug/
  - unmount without server restart, yey
- ensure we are backwards compatible
  - e.g. render home.hbs for collection index if collection route is /
  - ensure you can access your configured permalink from the settings table with {globals.permalink}

### Render 503 if url service did not finish

- return 503 if the url service has not finished generating the resource urls

### Rewrite sitemaps

- we have rewritten the sitemaps "service", because the url generator does no longer happen on runtime
- we generate all urls on bootstrap
- the sitemaps service will consume created resource and router urls
- these urls will be shown on the xml pages
- we listen on url events
- we listen on router events
- we no longer have to fetch the resources, which is nice
  - the urlservice pre-fetches resources and emits their urls
- the urlservice is the only component who knows which urls are valid
- i made some ES6 adaptions
- we keep the caching logic -> only regenerate xml if there is a change
- updated tests
- checked test coverage (100%)

### Re-work usage of Url utility

- replace all usages of `urlService.utils.urlFor` by `urlService.getByResourceId`
  - only for resources e.g. post, author, tag
- this is important, because with dynamic routing we no longer create static urls based on the settings permalink on runtime
- adapt url utility
- adapt tests
2018-06-05 19:02:20 +02:00
kirrg001
5f5f0021db 🔥 Drop Node v4 Support
no issue

- support ends today
- see https://github.com/nodejs/Release
- removed `use strict`
2018-05-01 14:06:18 +02:00
kirrg001
02abe3862e Use bookshelf-relations for Permission model: roles
refs https://github.com/TryGhost/Ghost/pull/9592

- we add bookshelf-relations step by step if we need it
- with https://github.com/TryGhost/Ghost/pull/9592 we have rewritten the test env to use Bookshelf
  - this is important for our new url service
  - because the service is listening on model updates and updates the urls based on the model events
- so with moving to Bookshelf, we need any easy way to add relations
  - the test env inserts test fixtures
  - it adds permissions and each permission get's roles attached
  - `models.Permission.add({roles: [...]})
2018-04-25 12:27:39 +02:00
Katharina Irrgang
6a4af1f465
Rewrite url service (#9550)
refs https://github.com/TryGhost/Team/issues/65

We are currently work on dynamic routing (aka channels).
An important piece of this feature is the url service, which always knows the url of a resource at any time.
Resources can belong to collections or taxonomies, which can be defined in a [routing yaml file](https://github.com/TryGhost/Ghost/issues/9528). We are currently shipping portions, which will at end form the full dynamic routing feature.

### Key Notes

- each routing type (collections, taxonomies, static pages) is registered in order - depending on the yaml routes file configuration
- static pages are an internal concept - they sit at the end of the subscriber queue
- we make use of a temporary [`Channels2`](https://github.com/TryGhost/Ghost/pull/9550/files#diff-9e7251409844521470c9829013cd1563) file, which simulates the current static routing in Ghost (this file will be modified, removed or whatever - this is one of the next steps)
- two way binding: you can ask for a resource url based on the resource id, you can ask for the resource based on the url
- in theory it's possible that multiple resources generate the same url: we don't handle this with collision (because this is error prone), we handle this with the order of serving content. if you ask the service for a resource, which lives behind e.g. /test/, you will get the resource which is served
- loose error handling -> log errors and handle instead of throw error and do nothing (we log the errors with a specific code, so we can react in case there is a bug)
- the url services fetches all resources on bootstrap. we only fetch and keep a reduced set of attributes (basically the main body of a resource)
- the bootstrap time will decrease a very little (depending on the amount of resources you have in your database)
- we still offer the option to disable url preloading (in your config `disableUrlPreload: true`) - this option will be removed as soon as the url service is connected. You can disable the service in case you encounter a problem
- **the url service is not yet connected, we will connect the service step by step. The first version should be released to pre-catch bugs. The next version will add 503 handling if the url service is not ready and it will consume urls for resources.**


----

- the url service generates urls based on resources (posts, pages, users, tags)
- the url service keeps track of resource changes
- the url service keeps track of resource removal/insert
- the architecture:
  - each routing type is represented by a url generator
    - a routing type is a collection, a taxonomiy or static pages
  - a queue which ensures that urls are unique and can be owned by one url generator
    - the hierarchy of registration defines that
  - we query knex, because bookshelf is too slow
- removed old url service files + logic
- added temp channels alternative (Channels2) -> this file will look different soon, it's for now the temporary connector to the url service. Also the name of the file is not optimal, but that is not really important right now.
2018-04-17 11:29:04 +02:00
Katharina Irrgang
7273786459
Fetch relations by default when insert/updating posts (#9568)
no issue

- required for model events
- otherwise you won't receive a full data set
  - in worst case you have to re-fetch the post
- required for the url service
  - the url service always needs relations (authors,tags) to be able to generate the url properly

@IMPORTANT
- no API change, we still return what you are asking for
  - we first edit/add the resource
  - then we fetch the data with the API options
  - @TODO: this can be optimised and will improve performance
    	   picking/selecting it from the insert/update response
- this is an internal change
2018-04-15 12:12:20 +02:00
kirrg001
aecca28257 Optimised emitChange for destroyed resources
no issue

- see comment in code base
2018-04-06 19:10:59 +02:00
Katharina Irrgang
fb79f24316
Fixed model events and transactions (#9524)
no issue

- if multiple queries run in a transaction, the model events are triggered before the txn finished
- if the txn rolls back, the events are anyway emitted
- the events are triggered too early
- solution:
  - `emitChange` needs to detect that a transaction is happening
  - it listens on a txn event to determine if events should be triggered
2018-04-06 18:19:45 +02:00
kirrg001
5928a5b240 Extended check for updated_at on model update
no issue

- ensure the schema type has a key `updated_at`, otherwise ignore
2018-04-05 18:51:58 +02:00
kirrg001
da80019aca Removed taking care of bookshelf's changed model keys
no issue

- this is no longer needed for now
- it was anyway a little bit ugly to modify bookshelf's `changed` object
- if we want to change something about figuring out if a model has changed (including relations)
  -> we probably need to override bookshelf
2018-04-05 18:51:58 +02:00
kirrg001
853b518a51 Sanitize incoming model relation data
refs #9548

- we always receive date strings from the client in ISO format
- we ensure that we transform these strings into JS dates for comparison
- when the client sends relations, we need to ensure that relations are checked as well
- will only work for the post model for now, because this is the only model which uses `bookshelf-relations`
- added unit tests
- removed some model tests, which do the same
2018-04-05 18:51:58 +02:00
kirrg001
5c5ecfd61d A bit of ES6 for models/base/index.js
no issue
2018-04-05 18:51:58 +02:00
Katharina Irrgang
40d0a745df Multiple authors (#9426)
no issue

This PR adds the server side logic for multiple authors. This adds the ability to add multiple authors per post. We keep and support single authors (maybe till the next major - this is still in discussion)

### key notes

- `authors` are not fetched by default, only if we need them
- the migration script iterates over all posts and figures out if an author_id is valid and exists (in master we can add invalid author_id's) and then adds the relation (falls back to owner if invalid)
- ~~i had to push a fork of bookshelf to npm because we currently can't bump bookshelf + the two bugs i discovered are anyway not yet merged (https://github.com/kirrg001/bookshelf/commits/master)~~ replaced by new bookshelf release
- the implementation of single & multiple authors lives in a single place (introduction of a new concept: model relation)
- if you destroy an author, we keep the behaviour for now -> remove all posts where the primary author id matches. furthermore, remove all relations in posts_authors (e.g. secondary author)
- we make re-use of the `excludeAttrs` concept which was invented in the contributors PR (to protect editing authors as author/contributor role) -> i've added a clear todo that we need a logic to make a diff of the target relation -> both for tags and authors
- `authors` helper available (same as `tags` helper)
- `primary_author` computed field available
- `primary_author` functionality available (same as `primary_tag` e.g. permalinks, prev/next helper etc)
2018-03-27 15:16:15 +01:00
Katharina Irrgang
7c6f690eb5 🐛 Fixed updated_at not being updated (#9532)
closes #9520

- it contains a dependency bump of the latest Bookshelf release
- Bookshelf introduced a bug in the last release
  - see https://github.com/bookshelf/bookshelf/pull/1583
  - see https://github.com/bookshelf/bookshelf/pull/1798
- this has caused trouble in Ghost
  - the `updated_at` attribute was not automatically set anymore

---

The bookshelf added one breaking change: it's allow to pass custom `updated_at` and `created_at`.
We already have a protection for not being able to override the `created_at` date on update.
We had to add another protection to now allow to only change the `updated_at` property.
You can only change `updated_at` if you actually change something else e.g. the title of a post.

To be able to implement this check i discovered that Bookshelfs `model.changed` object has a tricky behaviour.
It remembers **all** attributes, which where changed, doesn't matter if they are valid or invalid model properties.
We had to add a line of code to avoid remembering none valid model attributes in this object.

e.g. you change `tag.parent` (no valid model attribute). The valid property is `tag.parent_id`.
     If you pass `tag.parent` but the value has **not** changed (`tag.parent` === `tag.parent_id`), it will output you `tag.changed.parent`. But this is wrong.
     Bookshelf detects `changed` attributes too early. Or if you think the other way around, Ghost detects valid attributes too late.
     But the current earliest possible stage is the `onSaving` event, there is no earlier way to pick valid attributes (except of `.forge`, but we don't use this fn ATM).
     Later: the API should transform `tag.parent` into `tag.parent_id`, but we are not using it ATM, so no need to pre-optimise.
     The API already transforms `post.author` into `post.author_id`.
2018-03-26 14:12:02 +01:00
Katharina Irrgang
95423ea8fa
Bump dependencies (#9513)
no issue

- knex@0.14.4
- bookshelf@0.13.0
- knex-migrator@3.1.4
- brute-knex@4feff38ad2
- bookshelf-relations@0.2.0

### Fixes for Bookshelf 0.13

- they introduced some breaking changes
- https://github.com/bookshelf/bookshelf/blob/master/CHANGELOG.md#breaking-changes
- adapt event handling in Ghost and in bookshelf-relations
2018-03-19 16:27:06 +01:00
Aileen Nowak
1da2eec915 🐛Fixed image properties to be reset to null after removal (#9432)
closes #9085

Fixes an issue, where the client sets image properties to `""` after deleting the image. This causes problems with the query filter (see https://github.com/TryGhost/GQL/issues/24), as they have to be `null`.

Added a check in the model layer saving method to set value  to `null`, when the property is empty.

Affected models and properties:
- `posts`:
	- `feature_image`
	- `og_image`
	- `twitter_image`
- `users`:
	- `profile_image`
	- `cover_image`
- `tags`:
	- `feature_image`
2018-03-05 09:10:27 +01:00
kirrg001
5a4dd6b792 Increased speed of importer
no issue

- change behaviour from updating user references after the actual import to update the user reference before the actual import
  - updating user references after the import is way less case intense
  - that was the initial decision for updating the references afterwards
  - but that does not play well with adding nested relations by identifier
- the refactoring is required for multiple authors
  - if we e.g. store invalid author id's, we won't be able to add a belongs-to-many relation for multiple authors
  - bookshelf-relations is generic and always tries to find a matching target before attching a model
  - invalid user references won't work anymore
- this change has a very good side affect
  - 17mb takes on master ~1,5seconds
    - on this branch it takes ~45seconds
  - also the memory usage is way lower and stabler
  - 40mb takes 1,6s (times out on master)
2018-02-20 09:56:45 +01:00
Katharina Irrgang
0aff9f33d9
Improved validation layer (#9427)
refs https://github.com/TryGhost/Ghost/issues/3658

- the `validateSchema` helper was a bit broken
  - if you add a user without email, you will receive a database error
  - but the validation error should catch that email is passed with null
- it was broken, because:
  - A: it called `toJSON` -> this can remove properties from the output (e.g. password)
  - B: we only validated fields, which were part of the JSON data (model.hasOwnProperty)
- we now differentiate between schema validation for update and insert
- fixed one broken import test
  - if you import a post without a status, it should not error
  - it falls back to the default value
- removed user model `onValidate`
  - the user model added a custom implementation of `onValidate`, because of a bug which we experienced (see https://github.com/TryGhost/Ghost/issues/3638)
  - with the refactoring this is no longer required - we only validate fields which have changed when updating resources
  - also, removed extra safe catch when logging in (no longer needed - unit tested)
- add lot's of unit tests to proof the code change
- always call the base class, except you have a good reason
2018-02-16 00:49:15 +01:00
Katharina Irrgang
c6a95c6478
Sorted out the mixed usages of include and withRelated (#9425)
no issue

- this commit cleans up the usages of `include` and `withRelated`.

### API layer (`include`)
- as request parameter e.g. `?include=roles,tags`
- as theme API parameter e.g. `{{get .... include="author"}}`
- as internal API access e.g. `api.posts.browse({include: 'author,tags'})`
- the `include` notation is more readable than `withRelated`
- and it allows us to use a different easier format (comma separated list)
- the API utility transforms these more readable properties into model style (or into Ghost style)

### Model access (`withRelated`)
- e.g. `models.Post.findPage({withRelated: ['tags']})`
- driven by bookshelf

---

Commits explained.

* Reorder the usage of `convertOptions`

- 1. validation
- 2. options convertion
- 3. permissions
- the reason is simple, the permission layer access the model layer
  - we have to prepare the options before talking to the model layer
- added `convertOptions` where it was missed (not required, but for consistency reasons)

* Use `withRelated` when accessing the model layer and use `include` when accessing the API layer

* Change `convertOptions` API utiliy

- API Usage
  - ghost.api(..., {include: 'tags,authors'})
  - `include` should only be used when calling the API (either via request or via manual usage)
  - `include` is only for readability and easier format
- Ghost (Model Layer Usage)
  - models.Post.findOne(..., {withRelated: ['tags', 'authors']})
  - should only use `withRelated`
  - model layer cannot read 'tags,authors`
  - model layer has no idea what `include` means, speaks a different language
  - `withRelated` is bookshelf
  - internal usage

* include-count plugin: use `withRelated` instead of `include`

- imagine you outsource this plugin to git and publish it to npm
- `include` is an unknown option in bookshelf

* Updated `permittedOptions` in base model

- `include` is no longer a known option

* Remove all occurances of `include` in the model layer

* Extend `filterOptions` base function

- this function should be called as first action
- we clone the unfiltered options
- check if you are using `include` (this is a protection which could help us in the beginning)
- check for permitted and (later on default `withRelated`) options
- the usage is coming in next commit

* Ensure we call `filterOptions` as first action

- use `ghostBookshelf.Model.filterOptions` as first action
- consistent naming pattern for incoming options: `unfilteredOptions`
- re-added allowed options for `toJSON`
- one unsolved architecture problem:
  - if you override a function e.g. `edit`
  - then you should call `filterOptions` as first action
  - the base implementation of e.g. `edit` will call it again
  - future improvement

* Removed `findOne` from Invite model

- no longer needed, the base implementation is the same
2018-02-15 10:53:53 +01:00
Katharina Irrgang
9ede5905f6
Reduce toJSON implementation: use the power of bookshelf (#9423)
refs #6103

- simplify `toJSON`
- `baseKey` was not used - have not find a single use case
- all the functionality of our `toJSON` is offered in bookshelf
- `omitPivot` does remove pivot elements from the JSON obj (bookshelf feature)
- `shallow` allows you to not return relations
- make use of `serialize`, see http://bookshelfjs.org/docs/src_base_model.js.html#line260
- fetching nested relations e.g. `users.roles` still works (unrelated to this refactoring)

> pick('shallow', 'baseKey', 'include', 'context')

We will re-add options validation in https://github.com/TryGhost/Ghost/pull/9427, but then with the official way: use `filterOptions`.

---

We return all fetched relations (pre-defined with `withRelated`) by default.
You can disable it with `shallow:true`.
2018-02-14 17:32:11 +01:00
kirrg001
c6f30a46d2 Avoid knex warning when destroying a user
no issue

- the warning is "Transaction was already complete"
- destroying a user happens in a transaction, but the event is not asynchronous
  - so we have to ensure that we don't operate on a finished transaction
2018-01-27 12:31:51 +01:00
kirrg001
b4f355f713 Removed the usages of this.forge(null, {context: options.context})
no issue

- refs fe461da110
- the access plugin was removed
- no need to pass `context` as parameter for `.forge`
2018-01-26 00:35:39 +01:00
kirrg001
fe461da110 Deleted bookshelf access plugin
refs #9127

- permission checks can happen everywhere in the code base
  - we would like to create a context class
- global access to `options.context.is(...)`
- please read more about the access plugin in #9127 section "Model layer and the access plugin".
- removed the plugin and use direct context checks
2018-01-25 17:54:28 +01:00
Katharina Irrgang
5b77f052d9
Update Notification improvements (#9123)
closes #5071

- Remove hardcoded notification in admin controller
  - NOTE: update check notifications are no longer blocking the admin rendering
  - this is one of the most import changes
  - we remove the hardcoded release message
  - we also remove adding a notification manually in here, because this will work differently from now on
    -> you receive a notification (release or custom) in the update check module and this module adds the notification as is to our database

- Change default core settings keys
  - remove displayUpdateNotification
    -> this was used to store the release version number send from the UCS
    -> based on this value, Ghost creates a notification container with self defined values
    -> not needed anymore

- rename seenNotifications to notifications
  -> the new notifications key will hold both
     1. the notification from the USC
     2. the information about if a notification was seen or not
  - this key hold only one release notification
  - and n custom notifications

- Update Check Module: Request to the USC depends on the privacy configuration
  - useUpdateCheck: true -> does a checkin in the USC (exposes data)
  - useUpdateCheck: false -> does only a GET query to the USC (does not expose any data)
  - make the request handling dynamic, so it depends on the flag
  - add an extra logic to be able to define a custom USC endpoint (helpful for testing)
  - add an extra logic to be able to force the request to the service (helpful for testing)

- Update check module: re-work condition when a check should happen
  - only if the env is not correct
  - remove deprecated config.updateCheck
  - remove isPrivacyDisabled check (handled differently now, explained in last commit)

- Update check module: remove `showUpdateNotification` and readability
  - showUpdateNotification was used in the admin controller to fetch the latest release version number from the db
  - no need to check against semver in general, the USC takes care of that (no need to double check)
  - improve readability of `nextUpdateCheck` condition

- Update check module: refactor `updateCheckResponse`
  - remove db call to displayUpdateNotification, not used anymore
  - support receiving multiple custom notifications
  - support custom notification groups
  - the default group is `all` - this will always be consumed
  - groups can be extended via config e.g. `notificationGroups: ['migration']`

- Update check module: refactor createCustomNotification helper
  - get rid of taking over notification duplication handling (this is not the task of the update check module)
  - ensure we have good fallback values for non present attributes in a notification
  - get rid of semver check (happens in the USC) - could be reconsidered later if LTS is gone

- Refactor notification API
  - reason: get rid of in process notification store
    -> this was an object hold in process
    -> everything get's lost after restart
    -> not helpful anymore, because imagine the following case
      -> you get a notification
      -> you store it in process
      -> you mark this notification as seen
      -> you restart Ghost, you will receive the same notification on the next check again
      -> because we are no longer have a separate seen notifications object
  - use database settings key `notification` instead
  - refactor all api endpoints to support reading and storing into the `notifications` object
  - most important: notification deletion happens via a `seen` property (the notification get's physically deleted 3 month automatically)
    -> we have to remember a seen property, because otherwise you don't know which notification was already received/seen

- Add listener to remove seen notifications automatically after 3 month
  - i just decided for 3 month (we can decrease?)
  - at the end it doesn't really matter, as long as the windows is not tooooo short
  - listen on updates for the notifications settings
  - check if notification was seen and is older than 3 month
  - ignore release notification

- Updated our privacy document
- Updated docs.ghost.org for privacy config behaviour
- contains a migration script to remove old settings keys
2018-01-09 15:20:00 +01:00
kirrg001
428008e63d Revert "🐛 Fixed importer duplicate detection for posts"
refs #8717

- we decided to not changing the current importer behaviour
- no slug duplication detection means, importing posts can result in duplicates
2018-01-03 13:34:15 +00:00
kirrg001
02bd71d0f5 🐛 Fixed importer duplicate detection for posts
closes #8717

- this is now required, because we run import queries sequentiell
- this code protects two cases:
  - you have duplicate slugs in the JSON file (the first get's inserted, the second get's ignored)
  - you have an existing slug in the database and you try to import the same slug, get's ignored
2018-01-03 00:07:41 +01:00
kirrg001
740b247a80 Avoid moment deprecation warning when validating incoming dates
no issue

> Deprecation warning: value provided is not in a recognized ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions.
2017-12-14 17:04:06 +01:00
kirrg001
bb06a8426d Moved tokens, url safe and safe string utility to lib/security
refs #9178

- we could now also move any crypto usages to lib/security, but no priority
- the main goal is to tidy up our utils folder
2017-12-14 13:38:00 +01:00
kirrg001
f83cbf6117 Moved pipeline/sequence to lib/promise
refs #9178

- continue with killing our global utils folder
- i haven't found any better naming for lib/promise
- so, require single files for now
- instead of doing `promiseLib = require('../lib/promise')`
- we can optimise the requires later
2017-12-13 22:20:02 +01:00
Katharina Irrgang
397400b4f8
Moved visibility utility to static model fn (#9327)
refs #9178

- this logic belongs to a static model helper
- the visibility property is a model property, the knowledge about the visibility values belongs to the model
- rename the functions, so they make more sense
2017-12-13 13:19:51 +01:00
kirrg001
6f6c8f4521 Import lib/common only
refs #9178

- avoid importing 4 modules (logging, errors, events and i18n)
- simply require common in each file
2017-12-12 10:28:13 +01:00
kirrg001
ac2578b419 Moved errors,logging,i18n and events to lib/common
refs #9178
2017-12-12 10:28:13 +01:00
kirrg001
4265afe580 Moved utils/url.js to UrlService
refs #9178

- we have to take care that we don't end up in circular dependencies
  - e.g. API requires UrlService and UrlService needs to require the API (for requesting data)
- update the references
- we would like to get rid of the utils folder, this is/was the most complicated change
2017-12-11 20:05:33 +01:00
Katharina Irrgang
0bb81bb3c4
Bump knex-migrator to version 3.1.1 (#9199)
no issue

- adapt major changes of knex-migrator v3
- adapt migration scripts, simplify and add `down` (rollback) hook if possible
- clear Ghost cache after init hook (because of `knex-migrator migrate --init`)
- ensure db migrations work with the CLI
- updated troubleshooting guide (https://docs.ghost.org/v1/docs/troubleshooting#section-task-execute-is-not-a-function)

**For development only: Please ensure you run `npm i -g knex-migrator@latest` to update your global installation to v3. We always prefer the local installation, but v3 has modified and added binaries.**
2017-12-05 09:14:55 +01:00
Katharina Irrgang
dfd4afea19 Add bookshelf-relations (#9252)
no issue

- added https://github.com/TryGhost/bookshelf-relations as dependency
- remove existing tag handling

--- 

* Important: Ensure we trigger parent initialize function

- otherwise the plugin is unable to listen on model events
- important: event order for listeners is Ghost -> Plugin
- Ghost should be able to listen on the events as first instance
- e.g. be able to modify/validate relationships

* Fix tag validation

- we detect lower/update case slugs for tags manually
- this can't be taken over from the plugin obviously
- ensure we update the target model e.g. this.set('tags', ...)

* override base fn: `permittedAttributes`

- ensure we call the base
- put relations on top
- each relation is allowed to be passed
- the plugin will auto-unset any relations to it does not reach the database

* Ensure we run add/edit/delete within a transaction

- updating nested relationships requires sql queries
- all sql statements have to run in a single transaction to ensure we rollback everything if an error occurs
- use es6
2017-11-21 13:28:05 +00:00
Aileen Nowak
982a75d6be 🐛 Fixed slugs from exceeding db limit (#9251)
closes #8143

Fixed a potential issue (edge-case), where our generated and validated (in terms of check for existance and add a counter) would return a slug, that will exceed the maximum length of the slug fields (191 chars).

This is mostly possible for the post title, which can be 255 chars long and would generate a slug with the same length. This would prevent the user from actually saving a post.

I tried first to determine the expected length for a slug that already exists, but decided that the **easier** and simplyfied implementation is to always cut a slug to **185 chars** (+ counter). This makes it easier to find duplicates and includes a possible high number of counts (edge-edge-case).

The slug will not be cut down to 185 chars if it's an import.
2017-11-21 14:21:22 +01:00
kirrg001
4b21fc1d59 Allow forUpdate for any model
no issue

- this was only supported for the Post Model until now
- locking should be possible for every resource depending on the use case
2017-11-14 10:22:09 +00:00
Hannah Wolfe
bcf5a1bc34
Switch to Eslint (#9197)
refs #9178

* Add eslint deps, remove old lint deps
* Add eslint config, remove old lint configs
* Config for server and tests are different
* Tweaked rules to suit us
* Fix linting in codebase - lots of indent changes.
* Fix a real broken test
2017-11-01 13:44:54 +00:00
Katharina Irrgang
5f44972d44 🐛 Fixed being able to store invalid date formats (#9090)
closes #9089
- use the current date any time a post is fetched if the database contains an invalid date
- raise an error any time an attempt is made to save an invalidate date via the API
2017-10-04 09:56:09 +01:00
Katharina Irrgang
42af268d1b 🎨 User is not allowed to add/modify certain fields (#9053)
no issue

- it's not allowed to change/add these attributes via the API
  - created_at = is only once set on adding the resource
  - created_by = is only once set on adding the resource
  - updated_by = is set on the server side when updating the model (based on who is logged in)
  - updated_at = is set on the server side when updating the model

* Revert the usage of the access rules plugin
2017-09-28 13:59:42 +01:00
Katharina Irrgang
3002747b68 Return dates from the database without milliseconds (#9054)
no issue

- we store dates without milliseconds in the database
- our test environment does not use our model layer to insert data, this is related to  https://github.com/TryGhost/Ghost/issues/7196
- so it can happen that the test env inserts unix timestamps instead of a formatted string
- e.g. adding data via the model layer (e.g. via the API) the format is always normalised to `YYYY-MM-DD HH:mm:ss`
- if we fetch the date from the database, we have a hook which sorts out knex returning different formats for dates
- this hook wraps the returned date into a UTC moment date, but adds the current milliseconds on top
- which can collide in tests when you have specific assertions
- use `startOf` to ignore milliseconds
- furthermore: remove the mentionings of `pg` (postgres)
2017-09-26 17:16:46 +01:00
Katharina Irrgang
af01f51204 🐛 Fixed returning roles for the public user resource (#9039)
no issue

- this bug fix affects all endpoints for the public user access
- we allowed fetching `roles` via the public api by accident
- see our docs: https://api.ghost.org/docs/users)
  - we only allow `count.posts`
- returning roles via the public api exposes too many details
- this was never intentional
2017-09-26 15:43:21 +01:00
Katharina Irrgang
e921c7a044 Revert "🐛 Fixed returning roles for the public user resource (#9039)" (#9062)
This reverts commit 217bc6914d.

- NOTE: will be released in the next minor release
2017-09-26 14:28:34 +01:00
Katharina Irrgang
217bc6914d 🐛 Fixed returning roles for the public user resource (#9039)
no issue

- this bug fix affects all endpoints for the public user access
- we allowed fetching `roles` via the public api by accident
- see our docs: https://api.ghost.org/docs/users)
  - we only allow `count.posts`
- returning roles via the public api exposes too many details
- this was never attentional
2017-09-25 11:18:23 +01:00
Katharina Irrgang
b003a6c173 🐛 fix transfer ownership (#8784)
closes #8781

- when the ownership get's transferred, the id of the new owner is not '1' anymore
- we previously added a database rule, which signalises if the blog is setup or not, see 827aa15757 (diff-7a2fe80302d7d6bf67f97cdccef1f71fR542)
- this database rule is based on the owner id being '1', which is wrong when you transfer ownership
- we should keep in mind, that the owner id being '1' is only the default Ghost setup, but it can change
- blog is setup if the owner is locked
2017-07-31 13:37:37 +04:00
Katharina Irrgang
d6aaf2dbc7 🎨 do not run model listeners on import (#8720)
no issue

- if you upload a huge import file, parallel operations can throw errors e.g. lock wait exceeds
- this can happen if multiple transactions run in parallel
- there is no need to run:
  1. the removal of active tokens on import, because imported users have no active session
  2. rescheduling logic on timezone, because importing scheduled posts works out of the box via the model layer (if a published date is detected and it's in the future, the post get's scheduled)
2017-07-21 09:58:58 +01:00
Katharina Irrgang
59d7302da5 🐛 import invalid dates (#8712)
closes #8703, closes #8015

- add sanitize fn to importer
- check wether an imported date is a valid date
- if not, print a warning
2017-07-20 11:24:23 +01:00
kirrg001
d4c74e74c4 🐛 fix unknown user id on deactivated event
no issue

- if you delete an active user, Ghost logs an error message (Ghost does not crash!)
- but the event logic is not triggered, that means we don't delete the users tokens
- token deletion happens on: suspend a user and delete a user
2017-07-18 18:20:10 +01:00
Katharina Irrgang
1f37ff6053 🎨 refactor the importer (#8473)
refs #5422

- we can support null titles after this PR if we want
- user model: fix getAuthorRole
- user model: support adding roles by name
- we support this for roles as well, this makes it easier when importing related user roles (because usually roles already exists in the database and the related id's are wrong e.g. roles_users)
- base model: support for null created_at or updated_at values
- post or tag slugs are always safe strings
- enable an import of a null slug, no need to crash or to cover this on import layer
- add new DataImporter logic
    - uses a class inheritance mechanism to achieve an easier readability and maintenance
    - schema validation (happens on model layer) was ignored
    - allow to import unknown user id's (see https://github.com/TryGhost/Ghost/issues/8365)
    - most of the duplication handling happens on model layer (we can use the power of unique fields and errors from the database)
- the import is splitted into three steps:
  - beforeImport
    --> prepares the data to import, sorts out relations (roles, tags), detects fields (for LTS)
  - doImport
    --> does the actual import
  - afterImport
    --> updates the data after successful import e.g. update all user reference fields e.g. published_by (compares the imported data with the current state of the database)
- import images: markdown can be null
- show error message when json handler can't parse file
- do not request gravatar if email is null
- return problems/warnings after successful import
- optimise warnings in importer
- do not return warnings for role duplications, no helpful information
- error handler: return context information of error
- we show the affected json entries as one line in the UI
- show warning for: detected duplicated tag
- schema validation: fix valueMustBeBoolean translation
- remove context property from json parse error
2017-05-23 17:18:13 +01:00
Katharina Irrgang
4e2474a018 🎨 settings inconsistency (#8381)
no issue
- replace camelCase settings keys with underscore_case for consistency
- discussed here https://github.com/TryGhost/Ghost-Admin/pull/661#discussion_r112939982
2017-04-24 18:41:00 +01:00
Katharina Irrgang
76bd4fdef6 🙀 Image field naming & new img_url helper (#8364)
* 🙀  change database schema for images
    - rename user/post/tag images
    - contains all the required changes from the schema change

* Refactor helper/meta data
    - rename cover to cover_image
    - also rename default settings to match the pattern
    - rename image to profile_image for user
    - rename image to feature_image for tags/posts

* {{image}} >>> {{img_url}}
    - rename
    - change the functionality
    - attr is required
    - e.g. {{img_url feature_image}}

* gscan 1.0.0
    - update yarn.lock

* Update casper reference: 1.0-changes
    - see 5487b4da8d
2017-04-24 18:21:47 +01:00
Katharina Irrgang
c93f03b87e post update collision detection (#8328) (#8362)
closes #5599

If two users edit the same post, it can happen that they override each others content or post settings. With this change this won't happen anymore.

 Update collision for posts
- add a new bookshelf plugin to detect these changes
- use the `changed` object of bookshelf -> we don't have to create our own diff
- compare client and server updated_at field
- run editing posts in a transaction (see comments in code base)

🙀  update collision for tags
- `updateTags` for adding posts on `onCreated` - happens after the post was inserted
   --> it's "okay" to attach the tags afterwards on insert
   --> there is no need to add collision for inserting data
   --> it's very hard to move the updateTags call to `onCreating`, because the `updateTags` function queries the database to look up the affected post
- `updateTags` while editing posts on `onSaving` - all operations run in a transactions and are rolled back if something get's rejected

- Post model edit: if we push a transaction from outside, take this one

  introduce options.forUpdate
- if two queries happening in a transaction we have to signalise knex/mysql that we select for an update
- otherwise the following case happens:
  >> you fetch posts for an update
  >> a user requests comes in and updates the post (e.g. sets title to "X")
  >> you update the fetched posts, title would get overriden to the old one

use options.forUpdate and protect internal post updates: model listeners
- use a transaction for listener updates
- signalise forUpdate
- write a complex test

use options.forUpdate and protect internal post updates: scheduling
- publish endpoint runs in a transaction
- add complex test
- @TODO: right now scheduling api uses posts api, therefor we had to extend the options for api's
  >> allowed to pass transactions through it
  >> but these are only allowed if defined from outside {opts: [...]}
  >> so i think this is fine and not dirty
  >> will wait for opinions
  >> alternatively we have to re-write the scheduling endpoint to use the models directly
2017-04-19 14:53:23 +01:00
Katharina Irrgang
f2fd075075 🎨 fix previos/current date comparison (isEqual in bookshelf) (#8357)
no issue

- client dates are sent as ISO format (moment(..).format())
- server dates are in JS Date format
  >> when bookshelf fetches data from the database, all dates are transformed into JS dates
  >> see `parse` helper function
- Bookshelf updates the model with the client data via Bookshelf's `set` function
- therefor Bookshelf uses a simple `isEqual` function from lodash to detect changes
- .previous(attr) and .get(attr) return false
- that has the concequence that dates are always marked as "changed"
- internally we use our `hasDateChanged` if we have to compare previous/updated dates
- but Bookshelf is not in our control for this case
2017-04-19 10:59:09 +02:00
Katharina Irrgang
f7393686f8 🐛 UTC offset for scheduled posts when changing timezone (#8356)
no issue

- the UTC offset diff of the current and previous timezone must switch
- i have added more tests and more example case descriptions to understand why
2017-04-19 10:26:33 +02:00
Katharina Irrgang
59a8911830 🐛 fix hasDateChanged (#8291)
no issue
- i don't know if this never worked or has worked and something changed in bookshelf
- but this fixes: saving the content (no change for published_at) of a scheduled post within the 2minutes window
- add `beforeWrite` option to hasDateChanged helper, see comment
- use previous for `beforeWrite` operations
- add a test and fix some other small issues in the scheduler tests
2017-04-06 17:49:59 +01:00
Katharina Irrgang
587ff6f026 🎨 change last_login to last_seen (#8259)
refs #8258

* 🎨  change last_login to last_seen

- rename the column
- a change in Ghost-Admin is required as well

* test utils: revert export examples

* revert line breaks
2017-04-05 20:45:55 +01:00
Katharina Irrgang
c9f551eb96 suspend user feature (#8114)
refs #8111 
- Ghost returns now all (active+none active) users by default
- protect login with suspended status
- test permissions and add extra protection for suspending myself
- if a user is suspended and tries to activate himself, he won't be able to proceed the login to get a new token
2017-03-13 12:03:26 +00:00
Katharina Irrgang
fa38257170 🐛 🎨 old accesstokens are not cleaned up (#8065)
closes #8035
- create auth/utils
- use authUtils.createTokens for all cases
- decrease the expiry of the old access token before creating a new one
2017-03-01 10:12:03 +00:00
Katharina Irrgang
7eb316b786 replace auto increment id's by object id (#7495)
* 🛠  bookshelf tarball, bson-objectid

* 🎨  schema changes

- change increment type to string
- add a default fallback for string length 191 (to avoid adding this logic to every single column which uses an ID)
- remove uuid, because ID now represents a global resource identifier
- keep uuid for post, because we are using this as preview id
- keep uuid for clients for now - we are using this param for Ghost-Auth

*   base model: generate ObjectId on creating event

- each new resource get's a auto generate ObjectId
- this logic won't work for attached models, this commit comes later

* 🎨  centralised attach method

When attaching models there are two things important two know

1. To be able to attach an ObjectId, we need to register the `onCreating` event the fetched model!This is caused by the Bookshelf design in general. On this target model we are attaching the new model.
2. We need to manually fetch the target model, because Bookshelf has a weird behaviour (which is known as a bug, see see https://github.com/tgriesser/bookshelf/issues/629). The most important property when attaching a model is `parentFk`, which is the foreign key. This can be null when fetching the model with the option `withRelated`. To ensure quality and consistency, the custom attach wrapper always fetches the target model manual. By fetching the target model (again) is a little performance decrease, but it also has advantages: we can register the event, and directly unregister the event again. So very clean code.

Important: please only use the custom attach wrapper in the future.

* 🎨  token model had overriden the onCreating function because of the created_at field

- we need to ensure that the base onCreating hook get's triggered for ALL models
- if not, they don't get an ObjectId assigned
- in this case: be smart and check if the target model has a created_at field

* 🎨  we don't have a uuid field anymore, remove the usages

- no default uuid creation in models
- i am pretty sure we have some more definitions in our tests (for example in the export json files), but that is too much work to delete them all

* 🎨  do not parse ID to Number

- we had various occurances of parsing all ID's to numbers
- we don't need this behaviour anymore
- ID is string
- i will adapt the ID validation in the next commit

* 🎨  change ID regex for validation

- we only allow: ID as ObjectId, ID as 1 and ID as me
- we need to keep ID 1, because our whole software relies on ID 1 (permissions etc)

* 🎨  owner fixture

- roles: [4] does not work anymore
- 4 means -> static id 4
- this worked in an auto increment system (not even in a system with distributed writes)
- with ObjectId we generate each ID automatically (for static and dynamic resources)
- it is possible to define all id's for static resources still, but that means we need to know which ID is already used and for consistency we have to define ObjectId's for these static resources
- so no static id's anymore, except of: id 1 for owner and id 0 for external usage (because this is required from our permission system)
- NOTE: please read through the comment in the user model


* 🎨  tests: DataGenerator and test utils

First of all: we need to ensure using ObjectId's in the tests. When don't, we can't ensure that ObjectId's work properly.
This commit brings lot's of dynamic into all the static defined id's.
In one of the next commits, i will adapt all the tests.

* 🚨  remove counter in Notification API

- no need to add a counter
- we simply generate ObjectId's (they are auto incremental as well)
- our id validator does only allow ObjectId as id,1 and me

* 🎨  extend contextUser in Base Model

- remove isNumber check, because id's are no longer numbers, except of id 0/1
- use existing isExternalUser
- support id 0/1 as string or number

*   Ghost Owner has id 1

- ensure we define this id in the fixtures.json
- doesn't matter if number or string

* 🎨  functional tests adaptions

- use dynamic id's

* 🎨  fix unit tests

* 🎨  integration tests adaptions

* 🎨  change importer utils

- all our export examples (test/fixtures/exports) contain id's as numbers
- fact: but we ignore them anyway when inserting into the database, see https://github.com/TryGhost/Ghost/blob/master/core/server/data/import/utils.js#L249
- in 0e6ed957cd (diff-70f514a06347c048648be464819503c4L67) i removed parsing id's to integers
- i realised that this ^ check just existed, because the userIdToMap was an object key and object keys are always strings!
- i think this logic is a little bit complicated, but i don't want to refactor this now
- this commit ensures when trying to find the user, the id comparison works again
- i've added more documentation to understand this logic ;)
- plus i renamed an attribute to improve readability

* 🎨  Data-Generator: add more defaults to createUser

- if i use the function DataGenerator.forKnex.createUser i would like to get a full set of defaults

* 🎨  test utils: change/extend function set for functional tests

- functional tests work a bit different
- they boot Ghost and seed the database
- some functional tests have mis-used the test setup
- the test setup needs two sections: integration/unit and functional tests
- any functional test is allowed to either add more data or change data in the existing Ghost db
- but what it should not do is: add test fixtures like roles or users from our DataGenerator and cross fingers it will work
- this commit adds a clean method for functional tests to add extra users

* 🎨  functional tests adaptions

- use last commit to insert users for functional tests clean
- tidy up usage of testUtils.setup or testUtils.doAuth

* 🐛  test utils: reset database before init

- ensure we don't have any left data from other tests in the database when starting ghost

* 🐛  fix test (unrelated to this PR)

- fixes a random failure
- return statement was missing

* 🎨  make changes for invites
2016-11-17 09:09:11 +00:00
Katharina Irrgang
48387e4ffd 🎨 tidy up static id (owner, internal, external) usages (#7675)
refs #7494, refs #7495 

This PR is an extracted clean up feature of #7495.
We are using everywhere static id checks (userId === 0 or userId === 1).
This PR moves the static values into the Base model.
This makes it 1. way more readable and 2. we can change the id's in a central place.

I changed the most important occurrences - no tests are touched (yet!).

The background is: when changing from auto increment id (number) to ObjectId's (string) we still need to support id 1 and 0, because Ghost relies on these two static id's.
I would like to support using both: 0/1 as string and 0/1 as number.

1 === owner/internal
0 === external

Another important change:
User Model does not longer define the contextUser method, because i couldn't find a reason?
I looked in Git history, see 6e48275160
2016-11-09 15:01:07 +00:00
Katharina Irrgang
ca7b5643d5 🎨 more clean code in User Model (#7572)
* 🎨  do not call generateSlug twice for User.setup

* 🎨  call generatePasswordHash onSaving only

- now we can add defaults to User Model
- it was not possible before because add User model did the following:
  1. validate password length
  2. hash password manually
  3. call ghostBookshelf.Model.add and THEN bookshelf defaults fn gets triggered
- call generatePasswordHash in onSaving hook for all use case
- add more tests to user model, juhu
2016-10-14 18:24:38 +01:00
Katharina Irrgang
8cd5d9f6fe 🎨 register events in base model (#7560)
refs #7432

- all models implemented it's own initialize fn to register events
- we can register all events in the base model
- important: we only listen on the event, if the model has defined a hook for it
- this is just a small clean up PR
- register more bookshelf events
2016-10-14 13:37:01 +01:00
Katharina Irrgang
869a35c97d migrations: seeding is part of init db task (#7545)
* 🎨  move heart of fixtures to schema folder and change user model

- add fixtures.json to schema folder
- add fixture utils to schema folder
- keep all the logic!

--> FIXTURE.JSON
- add owner user with roles

--> USER MODEL
- add password as default
- findAll: allow querying inactive users when internal context (defaultFilters)
- findOne: do not remove values from original object!
- add: do not remove values from original object!

* 🔥  remove migrations key from default_settings.json

- this was a temporary invention for an older migration script
- sephiroth keep alls needed information in a migration collection

* 🔥   add code property to errors

- add code property to errors
- IMPORTANT: please share your opinion about that
- this is a copy paste behaviour of how node is doing that (errno, code etc.)
- so code specifies a GhostError

* 🎨  change error handling in versioning

- no need to throw specific database errors anymore (this was just a temporary solution)
- now: we are throwing real DatabaseVersionErrors
- specified by a code
- background: the versioning unit has not idea about seeding and population of the database
- it just throws what it knows --> database version does not exist or settings table does not exist

* 🎨  sephiroth optimisations

- added getPath function to get the path to init scripts and migration scripts
- migrationPath is still hardcoded (see TODO)
- tidy up database naming to transacting

*   migration init scripts are now complete

- 1. add tables
- 2. add fixtures
- 3. add default settings

* 🎨  important: make bootup script smaller!

- remove all TODO'S except of one
- no seeding logic in bootup script anymore 🕵🏻

*   sephiroth: allow params for init command

- param: skip (do not run this script)
- param: only (only run this script)
- very simple way

* 🎨  adapt tests and test env

- do not use migrate.populate anymore
- use sephiroth instead
- jscs/jshint

* 🎨  fix User model status checks
2016-10-12 16:18:57 +01:00
Katharina Irrgang
677502813e 🎨 replace process.env.NODE_ENV usages by config.get('env') (#7544)
closes #6629

- i had the case that in gravatar process.env.NODE_ENV was undefined and indexOf of undefined crashe my application
- so always use config to read current env
2016-10-11 13:53:52 +01:00
Hannah Wolfe
63094d3cc7 Move internal tags out of labs (#7519)
closes #6165

- internal tags has been in labs for a couple of months, we've fixed some bugs & are ready to ship
- removes all code that tests for the labs flag
- also refactors the various usage of the visibility filter into a single util
- all the tests still pass!!!
- this marks #6165 as closed because I think the remaining UI tasks will be handled as part of a larger piece of work
2016-10-10 09:51:03 +01:00
Katharina Irrgang
d81bc91bd2 Error creation (#7477)
refs #7116, refs #2001

- Changes the way Ghost errors are implemented to benefit from proper inheritance
- Moves all error definitions into a single file
- Changes the error constructor to take an options object, rather than needing the arguments to be passed in the correct order.
- Provides a wrapper so that any errors that haven't already been converted to GhostErrors get converted before they are displayed.

Summary of changes:

* 🐛  set NODE_ENV in config handler
*   add GhostError implementation (core/server/errors.js)
  - register all errors in one file
  - inheritance from GhostError
  - option pattern
* 🔥  remove all error files
*   wrap all errors into GhostError in case of HTTP
* 🎨  adaptions
  - option pattern for errors
  - use GhostError when needed
* 🎨  revert debug deletion and add TODO for error id's
2016-10-06 13:27:35 +01:00
Katharina Irrgang
1882278b5b 🎨 configurable logging with bunyan (#7431)
- 🛠  add bunyan and prettyjson, remove morgan

-   add logging module
  - GhostLogger class that handles setup of bunyan
  - PrettyStream for stdout

-   config for logging
  - @TODO: testing level fatal?

-   log each request via GhostLogger (express middleware)
  - @TODO: add errors to output

- 🔥  remove errors.updateActiveTheme
  - we can read the value from config

- 🔥  remove 15 helper functions in core/server/errors/index.js
  - all these functions get replaced by modules:
    1. logging
    2. error middleware handling for html/json
    3. error creation (which will be part of PR #7477)

-   add express error handler for html/json
  - one true error handler for express responses
  - contains still some TODO's, but they are not high priority for first implementation/integration
  - this middleware only takes responsibility of either rendering html responses or return json error responses

- 🎨  use new express error handler in middleware/index
  - 404 and 500 handling

- 🎨  return error instead of error message in permissions/index.js
  - the rule for error handling should be: if you call a unit, this unit should return a custom Ghost error

- 🎨  wrap serve static module
  - rule: if you call a module/unit, you should always wrap this error
  - it's always the same rule
  - so the caller never has to worry about what comes back
  - it's always a clear error instance
  - in this case: we return our notfounderror if serve static does not find the resource
  - this avoid having checks everywhere

- 🎨  replace usages of errors/index.js functions and adapt tests
  - use logging.error, logging.warn
  - make tests green
  - remove some usages of logging and throwing api errors -> because when a request is involved, logging happens automatically

- 🐛  return errorDetails to Ghost-Admin
  - errorDetails is used for Theme error handling

- 🎨  use 500er error for theme is missing error in theme-handler

- 🎨  extend file rotation to 1w
2016-10-04 16:33:43 +01:00
kirrg001
b79a18ca8f 🎨 Separate invites from user
refs #7420
- remove invite logic from user
- add invite model and adapt affected logic for inviting team members
2016-09-26 11:08:43 +02:00
kirrg001
0ae0a0b490 🎨 change how we get and set config
refs #6982
- a replace for all config usages
- always use config.get or config.set
- this a pure replacement, no logic has changed

[ci skip]
2016-09-20 15:59:34 +01:00
kirrg001
0b2940d587 🎨 add getSubdir and getProtectedSlugs
refs #6982

- do not add the subdir property as static config value
- create functions to read subdir and protected slugs on runtime
2016-09-20 15:59:34 +01:00
Katharina Irrgang
9349e99e54 🐛 fix delete by author as transaction (#7145)
closes #7137

Deleting the content from the database runs in a transaction. see
https://github.com/TryGhost/Ghost/blob/master/core/server/api/users.js#L390

`destroyByAuthor` is one of the operations we trigger to delete all the conent, see https://github.com/TryGhost/Ghost/blob/master/core/server/models/post.js#L647

The post model has a specific hook for deleting content to delete the relations as well, see https://github.com/TryGhost/Ghost/blob/master/core/server/models/post.js#L122

This hook is part of the transaction. But the `options` are ignored. `(model/*, attr, options*/)` 
We use the `options` to forward the transaction reference, which we need to pass into the bookshelf queries. So `return model.load('tags').call('related', 'tags').call('detach')` does not forward the transaction and that's why it stucks when deleting the content.
2016-09-19 14:45:36 +01:00
Vijay Kandy
ffd3ec563a fix: "url" field is undefined when restricting returned fields (#7089)
closes #6625

- "url" and "author" fields depend on {id, published_at, slug, author_id} to construct post url.
- implemented a generic solution by defining defaultColumnsToFetch() in
  base class for models.
- findPage() calls defaultColumnsToFetch() before loading models
- results are transformed by filtering out additional properties to return just the requested fields
- Added a test case to check for url and author fields
- Renamed allColumns as requestedColumns and used _.map instead of Promise.map
2016-07-18 22:21:47 +02:00
Katharina Irrgang
8c6ba47b0e fix: offset bug when switching timezones (#7101)
no issue
2016-07-15 17:44:22 +01:00
Katharina Irrgang
416c215018 change default order of posts (#7096)
closes #6932
- new default order of posts: scheduled, draft, published
- invent orderDefaultRaw fn for each model
- each model is able to create a default raw order query
- separate count and fetch query for fetchPage, because the count query where group/order statements attached
2016-07-15 11:04:10 +01:00
“kirrg001”
1421c92ba5 post-scheduling
refs #6413
- PUT endpoint to publish a post/page for the scheduler
- fn endpoint to get all scheduled posts (with from/to query params) for the scheduler
- hardcoded permission handling for scheduler client
- fix event bug: unscheduled
- basic structure for scheduling
- post scheduling basics
- offer easy option to change adapter
- integrate the default scheduler adapter
- update scheduled posts when blog TZ changes
- safety check before scheduler can publish a post (not allowed to publish in the future or past)
- add force flag to allow publishing in the past
- invalidate cache header for /schedules/posts/:id
2016-06-14 10:52:13 +02:00
Austin Burdine
44537bd15f deps: lodash@4.13.1
closes #6911
- update lodash to v4
- remove lodash.tostring override
- remove lodash from greenkeeper ignore
2016-06-11 13:13:55 -06:00
Austin Burdine
aed8c0800e internal tags feature
refs #6165
- change behavior to use 'visibility' property
- finish out client & server-side behavior
- add tests
2016-06-11 09:12:04 -06:00
Hannah Wolfe
369fd2c6bd Initial work on internal tags feature
refs #6165
2016-06-10 13:32:56 -06:00
kirrg001
16fc0d29bf feature: when timezone changes, reschedule all posts
closes #6406
- created listeners.js connector
- merged listeners.js with events.js (in models/base)
- set a post to draft when published_at would be in the past
- reschedule a post when published_at would be in the future
2016-06-10 14:58:19 +02:00
cobbspur
9430f5fa51 Add token added event, update last_login
closes #6845

- emit token.added event when new access token is created
- update last_login for user token.added event is triggered
2016-06-09 09:17:06 +01:00
Hannah Wolfe
80a79172c4 Revert "Revert "Force UTC at process level"" 2016-06-03 09:06:18 +01:00
Hannah Wolfe
78e693f469 Revert "Force UTC at process level" 2016-06-02 14:38:02 +01:00
kirrg001
ec176c243a Force UTC at process level
issues #6406 #6399
- all dates are stored as UTC with this commit
- use moment.tz.setDefault('UTC')
- add migration file to recalculate local datetimes to UTC
- store all dates in same format into our three supported databases
- add option to remeber migrations inside settings (core)
- support DST offset for migration
- ensure we force UTC in test env
- run whole migration as transaction
- extend: Settings.findOne function
2016-06-02 13:23:09 +02:00
Sebastian Gierlinger
01ae7ae49f Subscribers: Model, API & CSV import/export
- subscriber model
- subscriber app updates
- subscriber end points
- import/export CSV
- added headers to export file
- added dynamic email field detection for import
- returns stats object after CSV import
- mask error message from DB
2016-05-11 10:28:10 +02:00
Katharina Irrgang
f644d99460 add small permission improvements
no issue
- do not check client type in auth middleware
- offer filtering for findAll function in base
- add isInternalContext to base model
2016-05-08 14:22:55 +02:00
Jason Williams
b69477e838 Return computed columns when used in "columns"
Closes #6625
- Adds a failing test for not returning computed columns as well
  as for the bookshelf bug where extra columns passed into a fetch
  will result in the model having an extra "quoted" column.
- Filter model attributes for passing into "fetch" but used the
  entire list of columns for `toJSON`.
2016-04-11 09:58:07 -05:00
Joerg Henning
4a3d8c7ae8 Remove sanitize() from model/base because clean code 2016-04-03 08:17:17 +08:00
Jason Williams
fe13503470 Refactor authentication API into pipeline format
Refs #5508
2016-03-07 11:53:54 -06:00
Hannah Wolfe
1c85650108 Move db connection out of config
refs #5047

- database connections are not configuration
2016-02-12 13:56:23 +00:00
rfpe
7abcc43907 Harvest server side strings
closes #5617
- Replace all hard-coded server-side strings with i18n translations
2015-12-19 12:12:16 +01:00
Hannah Wolfe
d4c8b69673 Support ordering by count
refs #6009

- super quick and dirty way to support ordering by counts
- @TODO refactor :)
2015-11-22 17:18:30 +00:00
Hannah Wolfe
2aa16514a3 Rename post_count to count.posts
refs  #6009

- This is a straight rename, no functionality is added
- The dot syntax requires pre/post processing to convert the name
- This PR also includes several updates to the tests, as they weren't being run as part of Travis!
2015-11-20 14:59:58 +00:00
Hannah Wolfe
f8d9af1010 Add some debugging tools to filters
- pass debug: true to the API to get some useful debug output
- does not work in production mode

Note: I have added these lines back in so many times in the past month or so so that I could
figure out what was happening, I figured everyone else might find them useful.

TODO: use a proper logging method dependent on env
2015-11-18 19:17:06 +00:00
Hannah Wolfe
6a0f1cf231 Filter plugin with enforce/default logic
refs #5614, #5943

- adds a new 'filter' bookshelf plugin which extends the model
- the filter plugin provides handling for merging/combining various filters (enforced, defaults and custom/user-provided)
- the filter plugin also handles the calls to gql
- post processing is also moved to the plugin, to be further refactored/removed in future
- adds tests showing how filter could be abused prior to this commit
2015-11-17 10:39:44 +00:00
Sebastian Gierlinger
1b17456f5b Merge pull request #6067 from ErisDS/gql-update
deps: ghost-gql@0.0.3
2015-11-16 16:00:55 +01:00
Hannah Wolfe
a3bd00d978 deps: ghost-gql@0.0.3
- adds test for nested null/not null query issue
- make use of new findStatement tool
2015-11-16 14:44:48 +00:00
Hannah Wolfe
666a616551 Add access rules bookshelf plugin
refs #5614

- change isPublicContext to detectPublicContext
  - behaviour now expands the context object out
  - this is a bit of a sideeffect, but this is the simplest change
    that makes it possible to use the context in the model layer without
    significant wider changes
- add new access rules plugin
  - takes a context object as part of `forge()` & caches it on the model instance
  - provides helper functions for testing access rules later on
2015-11-16 12:24:01 +00:00
Hannah Wolfe
4dac01cbf9 Refactor old processOptions/where to use GQL JSON
refs #5943

- no longer assume the options in processOptions are set
- set where to a new GQL JSON-like statement object
- rather than setting options, add statements which can be understood by knexify
- pass the statements through knexify to build the query
2015-11-12 17:24:09 +00:00
Hannah Wolfe
8edf382b40 Move bookshelf plugins into own folder 2015-11-11 19:40:12 +00:00
Hannah Wolfe
e0a6d027c8 Move cross-table api counts into plugin
refs #6009, #5615

- minimal refactor to remove the addition of count from pagination and other various points
- create a include count plugin that overrides fetch and fetchAll
- this ensures that counts get added at the right points
2015-11-03 14:09:38 +00:00
Hannah Wolfe
ea402218d3 Pagination cleanup & improvements
no issue

- switching from using fetch to fetch all means some code can be removed from the fetchPage method
- updating tests to reflect cleaner code
- ensure coverage is at 100%
2015-11-03 11:01:48 +00:00
vdemedes
6db41584e7 Add order parameter
refs #5602
- add "order" to default browse options
- parse order parameter in Base model
- accept "order" option in Post, User and Tag models
- add tests for posts order
- add tests for tags order
- add tests for users order
2015-10-28 14:14:03 +01:00
Hannah Wolfe
b8a3415726 Remove featured, tag, author & role API params
refs #5943

- removed featured, tag and author parameters from posts API
   - featured was only used in tests
- removed role filter from users API
   - role was only used in tests
- fixed up the tests, skipping those that don't quite work yet
2015-10-27 10:53:51 +00:00
Sebastian Gierlinger
28871d3f4d Merge pull request #5978 from ErisDS/filter-param
Add the filter parameter
2015-10-22 16:42:13 +02:00
Hannah Wolfe
b5cebb9ec6 Add filter parameter using GQL
refs #5604, refs #5463

- deps: ghost-gql@0.0.2
- adds code to wire up the filtering to a paginated query
- updated pagination plugin count query to use 'distinct' so it's more robust
- rename paginationUtils.query to addLimitAndOffset to be more explicit and make the code clearer
- add a new 'advanced browsing spec' set of tests for tracking these features as they are built out
2015-10-22 11:29:05 +01:00
cobbspur
372907890f Remove invalid fields prior to fetch
closes #5601

- Remove invalid fields prior to fetch
- Adds initial tests for fields
2015-10-21 18:20:09 +01:00