* 🐛 Dynamic Routing Beta: Shortform `author.foo` is not allowed
refs #9601
- otherwise you get access to {{author}} in the theme, which is deprecated & causes errors
- recommend not using {{author}} as data longform name in the yaml validator
- (internal usage is still allowed)
- also warn against using reserved data key names like filter, resource, limit etc - maybe remove this restriction later but seems like a sensible validation right now.
refs #9601
- when using the short form `data: tag.welcome` the redirect is enabled by default
- /tag/welcome/ will redirect to the channel/collection which makes use of the data key
- you can disable the redirect by using the long form
e.g. data:
tag:
resource: tags
type: read
slug: welcome
redirect: false
refs #9742
- removed usage of single permalink setting
- with dynamic routing this configuration does no longer makes sense
- because you can configure your permalinks in the routes.yaml
- furthermore you can have multiple collections with multiple permalinks
- removed @blog.permalinks
- do not export permalink setting
- do not import permalink setting
- permalink setting UI will be removed soon
- get rid of {globals.permalink} completely
- remove yaml in-built migration
- do not expose settings.permalinks via the private API
- do not allow to edit this setting
- keep phyiscal value in case a blog needs to rollback from v2 to v1
- sorted out when the routers should be created
- ensure routes.yaml file doesn't get validated before Ghost is fully ready to start
refs #9742, refs #9724
- handle König Editor format for 2.0
- adapted importer to be able to import 1.0 and 2.0 exports
- added migration scripts
- remove labs flag for Koenig
- migrate all old editor posts to new editor format
- ensure we protect the code against mobiledoc or html field being null
- ensure we create a blank mobiledoc structure if mobiledoc field is null (model layer)
- ensure you can fully rollback 2.0 to 1.0
- keep mobiledoc/markdown version 1 logic to be able to rollback (deprecated code)
refs #9742
- rebase against master updated some docs links again
- go over code base again and double check that all docs links are correct
- 2.0 will become the latest version on our readme pages
refs #9601
- on resource update/add we have cached mobiledoc, html etc
- we have to ensure we exclude the fields (same procdure happens on bootstrap)
- these excluded fields don't have to be cached
- otherwise memory usage is higher in general
- ensure we cache relations with a minimal field set on resource update/add
refs #9742
- Ghost 2.0 is coming
- all doc links in 1.0 must use concrete links e.g. docs.ghost.org/v1 or themes.ghost.org/v1.23.0/
- if we release Ghost 2.0, docs.ghost.org will show 2.0 docs
closes#9727
- this is a short term fix to proof that Disqus comments on preview pages no longer appear on other threads
- this is not a full solution to the problem
- the private API still returns /404/, which is right now inconsistent, but not critical in any way
- the url helper will now output the post preview url if you serve a draft/scheduled post
- this should register unique page urls at Disqus and ensure uniquness for threads
- i still don't understand why the cross posting happens at all, because we also pass an unique identifier to Disqus (the post ID)
- it could be that comments, which are added on the preview page, won't appear on the published urls, because
the published url !== preview url. I wasn't able to figure this out via testing or reading their docs
refs #9601
- this was already working for collections or channels
- but the `routeName` was not parsed for static routes
- ensure we push the route name into the context object
e.g. /about/: about
-> name of the route is "about"
no issue
- discovered while testing (issue doesn't exist)
- with dynamic routing we have introduced a bug
- we pre-generate urls and if your permalink contains a date (dated permalink),
we have to regenerate the urls, otherwise the dated urls do not respect your blog timezone
- collection router has to subscribe to the timezone event
- collection router must trigger an update on it's url generator if the timezone changes and the it's permalink is dated
- ensure we also update the urls on import
refs #9601
- replace jsonpath with [NQL](https://github.com/NexesJS/NQL)
- jsonpath was just a temporary solution (a short-term fix)
- with NQL we are able to filter collections more powerful in the near future
- NQL is not feature complete
- we still support `featured:true` for collections
refs #9601
- you can now use `rss:false`
- ability to define a custom rss url with a target template (+ content_type)
- ability to disable rss for channel or collection
refs #9601
- refactor architecture of routing so you can define a channel
- a channel is a different way of looking at your posts (a view)
- a channel does not change the url of a resource
Example channel
```
routes:
/worldcup-2018-russia/:
controller: channel
filter: tag:football18
data: tag.football18
```
- added ability to redirect resources to a channel/static route
- support templates for channels
- ensure we still support static routes (e.g. /about/: home)
- ensure pagination + rss works out of the box
refs #9601
- support data, limit and order for collections
- limit definition in routes.yaml is stronger than theme package.json limit configuration
- ensure we update hbs template options
refs #9601
- the home.hbs behaviour for the index collection (`/`) is hardcoded in Ghost
- we would like to migrate all existing routes.yaml files
- we only replace the file if the contents of the routes.yaml file equals the old routes.yaml format (with home.hbs as template)
- updated README of settings folder
- if we don't remove the home.hbs template from the default routes.yaml file, home.hbs will be rendered for any page of the index collection
- the backwards compatible behaviour was different
- only render home.hbs for page 1
- remember: the default routes.yaml file reflects how Ghost was working without dynamic routing
refs #9601
Example:
```
collections:
/podcast/:
permalink: /{slug}/
```
- the name of the collection is remembered as `routerName` (in the case above: "podcast")
- the name of the collection is important for two things
1. context value
2. template name
- the context value is available for specific theme helpers e.g. is helper, body_class helper
- we auto-lookup the collection name in your theme e.g. podcast.hbs
- this logic does not apply to static routes
- if you define templates on your collection, they are stronger than the collection name
refs #9601
- implementation of resource listener updated
- if you define two collections: `featured:true` (1) and `page:false` (2) you can run into the following bug:
- you create a published post (owned by (2))
- you change the status to featured
- still owned by (2), because the filter still matches (it's still not a static page)
- this adaption fixes the behaviour
- less smart logic, but less error prone
closes https://github.com/TryGhost/Ghost/issues/9674
- with dynamic routing the first collection get's the "index" context attached
- the index context signalises the main post listening route (first collection)
- this behaviour was present < 1.24 - we have to keep this behaviour
no issue
- this mock eat already too much of my/our time
- the idea of adding a knex mock was definitely a failed approach/try
- it's too much to maintaince and have not found a module which does this already
- we have to support any query format
- this is too crazy
- the idea was to use the knex mock for model unit tests, because if we want to unit test models we have to
run through bookshelf, because the whole model layer depends on bookshelf e.g. events
- for now we simply use the real database
- we could use the sqlite3 memory mode, but that would mean every unit test runs on sqlite3
- something to consider for later e.g. run unit tests on one matrix
- run the rest on another matrix for sqlite + mysql
closes#9675
- with dynamic routing we have introduced a breaking change, which we have overseen
- Ghost does not return absolute urls, that's why the clients need to concat the blog url and the resource url
- with 1.24.0 Ghost returned resource urls including the subdirectory
- this caused trouble for e.g. zapier or the preview feature in the admin client
- revert breaking change and ensure we only expose resource urls without subdirectory
no issue
- was introduced with dynamic routing beta: https://github.com/TryGhost/Ghost/releases/tag/1.24.0
- the slug param wasn't forwarded correctly
- you were not able to render a custom tag or author template e.g. `tag-news.hbs`
closes#9674
- the collection router had a hardcoded default context "home"
- this is wrong
- the context array get's automatically filled for the collection
- if you are serving a page e.g. /page/2/ -> it's "paged"
- if you are serving / -> it's "home"
- same for {{body_class}}, it outputs "home-template" on "/"
- this is the same behaviour as in 1.23.x
no issue
- reverse must happen once in the constructor
- otherwise we reverse the array on each request
- Ghost would randomly pick the first and then the second template
no issue
- there was a timing bug in Ghost
- we do operations in parallel on bootstrap
- 1) we fetch the resources as early as possible
- 2) we do all the rest (express bootstrapping, theme loading, router registration) etc.
- it can happen that (2) happens too slow and ends in the situation that the queue, which is responsible
to handle both parallel actions, does not wait for the routers and closes the event
- this is a short term fix
- i need to reconsider if there is a better long term fix
refs #9601
### Dynamic Routing
This is the beta version of dynamic routing.
- we had a initial implementation of "channels" available in the codebase
- we have removed and moved this implementation
- there is now a centralised place for dynamic routing - server/services/routing
- each routing component is represented by a router type e.g. collections, routes, static pages, taxonomies, rss, preview of posts
- keep as much as possible logic of routing helpers, middlewares and controllers
- ensure test coverage
- connect all the things together
- yaml file + validation
- routing + routers
- url service
- sitemaps
- url access
- deeper implementation of yaml validations
- e.g. hard require slashes
- ensure routing hierarchy/order
- e.g. you enable the subscriber app
- you have a custom static page, which lives under the same slug /subscribe
- static pages are stronger than apps
- e.g. the first collection owns the post it has filtered
- a post cannot live in two collections
- ensure apps are still working and hook into the routers layer (or better said: and register in the routing service)
- put as much as possible comments to the code base for better understanding
- ensure a clean debug log
- ensure we can unmount routes
- e.g. you have a collection permalink of /:slug/ represented by {globals.permalink}
- and you change the permalink in the admin to dated permalink
- the express route get's refreshed from /:slug/ to /:year/:month/:day/:slug/
- unmount without server restart, yey
- ensure we are backwards compatible
- e.g. render home.hbs for collection index if collection route is /
- ensure you can access your configured permalink from the settings table with {globals.permalink}
### Render 503 if url service did not finish
- return 503 if the url service has not finished generating the resource urls
### Rewrite sitemaps
- we have rewritten the sitemaps "service", because the url generator does no longer happen on runtime
- we generate all urls on bootstrap
- the sitemaps service will consume created resource and router urls
- these urls will be shown on the xml pages
- we listen on url events
- we listen on router events
- we no longer have to fetch the resources, which is nice
- the urlservice pre-fetches resources and emits their urls
- the urlservice is the only component who knows which urls are valid
- i made some ES6 adaptions
- we keep the caching logic -> only regenerate xml if there is a change
- updated tests
- checked test coverage (100%)
### Re-work usage of Url utility
- replace all usages of `urlService.utils.urlFor` by `urlService.getByResourceId`
- only for resources e.g. post, author, tag
- this is important, because with dynamic routing we no longer create static urls based on the settings permalink on runtime
- adapt url utility
- adapt tests
closes https://github.com/TryGhost/Ghost/issues/9620
- adjust the `deduplicateSubDir` function's regex to only match duplicate subdirectories when the `url` is only a path rather than full url or the duplicate match starts with a `/`
closesTryGhost/Support#426
refs TryGhost/gscan#106
needs TryGhost/gscan#107
GScan can return errors, which was not handled in our theme validator and caused Ghost to crash completely. GScan will now return an Ignition error when its not able to read the `.zip` file.
e. g.: `{"errors":[{"message":"Failed to read zip file","context":"tife.zip","errorType":"ValidationError","errorDetails":"invalid relative path: ../tife/"}]}`
refs https://github.com/TryGhost/Team/issues/65
- it's easier for the architecture if we read the setting files synchronously,
because the dynamic routing component is part of the express bootstrap and
the whole routing bootstrap is synchronously
- for now: we only read one file anyway
- it's for now easier to read the file synchronously, then i don't have to change
any existing express bootstrap architecture
refs https://github.com/TryGhost/Team/issues/65
- this is just the first optimisation regarding relative/absolute urls
- the full strike will happen when i start with the url utility re-write
- for now: there will be only one subscriber of url events -> the sitemaps service
- the sitemaps service outputs absolute urls
- we don't want to receive an url event and ask the url service again to get an absolute version of the url
refs https://github.com/TryGhost/Team/issues/65
- currently we generate a relative resource url
- if you configure a subdirectory, the urls have to respect that
- e.g. you configure `localhost:2368/blog`, your url results in e.g. `/blog/my-post/`
- this is not yet a critical bug, because the url service is not connected yet
- @TODO: consider absolute vs. relative urls in the url service
refs https://github.com/TryGhost/Team/issues/65
We are currently work on dynamic routing (aka channels).
An important piece of this feature is the url service, which always knows the url of a resource at any time.
Resources can belong to collections or taxonomies, which can be defined in a [routing yaml file](https://github.com/TryGhost/Ghost/issues/9528). We are currently shipping portions, which will at end form the full dynamic routing feature.
### Key Notes
- each routing type (collections, taxonomies, static pages) is registered in order - depending on the yaml routes file configuration
- static pages are an internal concept - they sit at the end of the subscriber queue
- we make use of a temporary [`Channels2`](https://github.com/TryGhost/Ghost/pull/9550/files#diff-9e7251409844521470c9829013cd1563) file, which simulates the current static routing in Ghost (this file will be modified, removed or whatever - this is one of the next steps)
- two way binding: you can ask for a resource url based on the resource id, you can ask for the resource based on the url
- in theory it's possible that multiple resources generate the same url: we don't handle this with collision (because this is error prone), we handle this with the order of serving content. if you ask the service for a resource, which lives behind e.g. /test/, you will get the resource which is served
- loose error handling -> log errors and handle instead of throw error and do nothing (we log the errors with a specific code, so we can react in case there is a bug)
- the url services fetches all resources on bootstrap. we only fetch and keep a reduced set of attributes (basically the main body of a resource)
- the bootstrap time will decrease a very little (depending on the amount of resources you have in your database)
- we still offer the option to disable url preloading (in your config `disableUrlPreload: true`) - this option will be removed as soon as the url service is connected. You can disable the service in case you encounter a problem
- **the url service is not yet connected, we will connect the service step by step. The first version should be released to pre-catch bugs. The next version will add 503 handling if the url service is not ready and it will consume urls for resources.**
----
- the url service generates urls based on resources (posts, pages, users, tags)
- the url service keeps track of resource changes
- the url service keeps track of resource removal/insert
- the architecture:
- each routing type is represented by a url generator
- a routing type is a collection, a taxonomiy or static pages
- a queue which ensures that urls are unique and can be owned by one url generator
- the hierarchy of registration defines that
- we query knex, because bookshelf is too slow
- removed old url service files + logic
- added temp channels alternative (Channels2) -> this file will look different soon, it's for now the temporary connector to the url service. Also the name of the file is not optimal, but that is not really important right now.
closes#9528
These code changes introduce a YAML parser which will load and parse YAML files from the `/content/settings` directory. There are three major parts involved:
1. `ensure-settings.js`: this fn takes care that on bootstrap, the supported files are present in the `/content/settings` directory. If the files are not present, they get copied back from our default files. The default files to copy from are located in `core/server/services/settings`.
2. `loader.js`: the settings loader reads the requested `yaml` file from the disk and passes it to the yaml parser, which returns a `json` object of the file. The settings loader throws an error, if the file is not accessible, e. g. because of permission errors.
3. `yaml-parser`: gets passed a `yaml` file and returns a `json` object. If the file is not parseable, it returns a clear error that contains the information, what and where the parsing error occurred (e. g. line number and reason).
- added a `get()` fn to settings services, that returns the settings object that's asked for. e. g. `settings.get('routes').then(()...` will return the `routes` settings.
- added a `getAll()` fn to settings services, that returns all available settings in an object. The object looks like: `{routes: {routes: {}, collections: {}, resources: {}}, globals: {value: {}}`, assuming that we have to supported settings `routes` and `globals`.
Further additions:
- config `contentPath` for `settings`
- config overrides for default `yaml` files location in `/core/server/services/settings`
**Important**: These code changes are in preparation for Dynamic Routing and not yet used. The process of copying the supported `yaml` files (in this first step, the `routes.yaml` file) is not yet activated.
no issue
This PR adds the server side logic for multiple authors. This adds the ability to add multiple authors per post. We keep and support single authors (maybe till the next major - this is still in discussion)
### key notes
- `authors` are not fetched by default, only if we need them
- the migration script iterates over all posts and figures out if an author_id is valid and exists (in master we can add invalid author_id's) and then adds the relation (falls back to owner if invalid)
- ~~i had to push a fork of bookshelf to npm because we currently can't bump bookshelf + the two bugs i discovered are anyway not yet merged (https://github.com/kirrg001/bookshelf/commits/master)~~ replaced by new bookshelf release
- the implementation of single & multiple authors lives in a single place (introduction of a new concept: model relation)
- if you destroy an author, we keep the behaviour for now -> remove all posts where the primary author id matches. furthermore, remove all relations in posts_authors (e.g. secondary author)
- we make re-use of the `excludeAttrs` concept which was invented in the contributors PR (to protect editing authors as author/contributor role) -> i've added a clear todo that we need a logic to make a diff of the target relation -> both for tags and authors
- `authors` helper available (same as `tags` helper)
- `primary_author` computed field available
- `primary_author` functionality available (same as `primary_tag` e.g. permalinks, prev/next helper etc)
no issue
- this commit cleans up the usages of `include` and `withRelated`.
### API layer (`include`)
- as request parameter e.g. `?include=roles,tags`
- as theme API parameter e.g. `{{get .... include="author"}}`
- as internal API access e.g. `api.posts.browse({include: 'author,tags'})`
- the `include` notation is more readable than `withRelated`
- and it allows us to use a different easier format (comma separated list)
- the API utility transforms these more readable properties into model style (or into Ghost style)
### Model access (`withRelated`)
- e.g. `models.Post.findPage({withRelated: ['tags']})`
- driven by bookshelf
---
Commits explained.
* Reorder the usage of `convertOptions`
- 1. validation
- 2. options convertion
- 3. permissions
- the reason is simple, the permission layer access the model layer
- we have to prepare the options before talking to the model layer
- added `convertOptions` where it was missed (not required, but for consistency reasons)
* Use `withRelated` when accessing the model layer and use `include` when accessing the API layer
* Change `convertOptions` API utiliy
- API Usage
- ghost.api(..., {include: 'tags,authors'})
- `include` should only be used when calling the API (either via request or via manual usage)
- `include` is only for readability and easier format
- Ghost (Model Layer Usage)
- models.Post.findOne(..., {withRelated: ['tags', 'authors']})
- should only use `withRelated`
- model layer cannot read 'tags,authors`
- model layer has no idea what `include` means, speaks a different language
- `withRelated` is bookshelf
- internal usage
* include-count plugin: use `withRelated` instead of `include`
- imagine you outsource this plugin to git and publish it to npm
- `include` is an unknown option in bookshelf
* Updated `permittedOptions` in base model
- `include` is no longer a known option
* Remove all occurances of `include` in the model layer
* Extend `filterOptions` base function
- this function should be called as first action
- we clone the unfiltered options
- check if you are using `include` (this is a protection which could help us in the beginning)
- check for permitted and (later on default `withRelated`) options
- the usage is coming in next commit
* Ensure we call `filterOptions` as first action
- use `ghostBookshelf.Model.filterOptions` as first action
- consistent naming pattern for incoming options: `unfilteredOptions`
- re-added allowed options for `toJSON`
- one unsolved architecture problem:
- if you override a function e.g. `edit`
- then you should call `filterOptions` as first action
- the base implementation of e.g. `edit` will call it again
- future improvement
* Removed `findOne` from Invite model
- no longer needed, the base implementation is the same
refs #9178
- this util uses the url services (!)
- moving this file into lib would not make sense right now
- that would mean a module requires first ../lib/url, which then requires ../services/url
- the url service definitely need a clean up 😃
refs #9178
- not 100% sure about this, but i think it makes right now the most sense
- we have already a url service and creating another lib/url is confusing at the moment
- i'll copy the last utility `makeAbsoluteUrls` to the url service for now
- see next commit for explanation (!)
refs #9178
- we have to take care that we don't end up in circular dependencies
- e.g. API requires UrlService and UrlService needs to require the API (for requesting data)
- update the references
- we would like to get rid of the utils folder, this is/was the most complicated change
no issue
Support for http://resthooks.org style webhooks that can be used with Zapier triggers. This can currently be used in two ways:
a) adding a webhook record to the DB manually
b) using the API with password auth and POSTing to /webhooks/ (this is private API so not documented)
⚠️ only _https_ URLs are supported in the webhook `target_url` field 🚨
- add `webhooks` table to store event names and target urls
- add `POST` and `DELETE` endpoints for `/webhooks/`
- configure `subscribers.added` and `subscribers.deleted` events to trigger registered webhooks
refs #9192, refs #5091
- Moved all url generation into generate-feed.js, so we can see as much data processing as possible in a single place.
- Refactored the way res.locals were used, to be more like how express uses them prior to rendering
- Removed a bunch of code & tests todo with context for RSS - I can't see any way that'd be used, unless we switched the rendering to use a template.
- moved the RSS rendering to be part of the service, not controller
- updated the tests significantly
Note: RSS generate-feed has a complete duplication of the code used in the excerpt helper in order to create an item description
refs #9192, refs #9178
After trying to progress with current implementation, it became clear that the route service can't control the boot sequence, because then we end up with circular dependencies between the route service and the channel service.
The route service now exposes:
- a siteRouter
- a way for apps to register routes.
- ParentRouter base class for other modules to use
- the registry
...
- moved the default route setup back to site/routes.js 🙈
- moved the parent channel router back to the channel service (this makes way more sense imo)
- this structure prevents circular dependencies
- split the registry out into it's own thing
- fixed-up various bits of tests and comments
- DEBUG will print a list of routes 🎉
refs #9192, #5091
- changed channels to use our new base class
- keep the flexible structure, so that channels can be reloaded
- I had to move the router into the route service otherwise we get circular dependencies
- Don't _really_ want to keep it like this - need a way to define base classes as shared
refs #9192, refs #5091, refs #9178
- moved channels from controllers to a service
- split out the parent router from the remaining individual router logic
- moved the tests to match
refs #5091, #9192, #9178
- Get the RSS module into a much better shape
- Controller -> /controllers/rss
- Remainder -> /services/rss
- Moved tests to match & updated requires
refs #9178
* Moved app handling code into services/apps
- Apps is a service, that allows for the App lifecycle
- /server/apps = contains internal apps
- /server/services/apps = contains code for managing/handling app life cycle, providing the proxy, etc
* Split apps service tests into separate files
* Moved internal app tests into test folders
- Problem: Not all the tests in apps were unit tests, yet they were treated like they were in Gruntfile.js
- Unit tests now live in /test/unit/apps
- Route tests now live in /test/functional/routes/apps
- Gruntfile.js has been updated to match
* Switch api.read usage for settingsCache
* Add tests to cover the basic App lifecycle
* Simplify some of the init logic
refs #9178
- Introduce the /services/ folder
- Move xmlrpc there
- Move slack there
- In slack: remove a usage of the settings API that should use settingsCache
- In slack: Simplify the tests
- Various tiny changes to move towards code consistency
